75 lines
14 KiB
Markdown
75 lines
14 KiB
Markdown
## Context
|
|
|
|
`apex-agent-mvp` (code-complete, unapplied) already produces a `Scene` for every Observe step: `core/models.py` defines `Scene { width, height, elements: [SceneElement] }` and `SceneElement { id, type, bounds, text, confidence, source }`, built by `vision/scene_builder.py` (planned rename: `perception/scene_builder.py` per `device-agent-runtime-foundation`) by fusing OCR output and the device's UI tree via bounding-box IoU. `runtime/planner.py` (`Planner.plan()`) is currently a stub that returns one hardcoded `describe_screen` step and then nothing — there is no real LLM call anywhere in the codebase yet. This change introduces the **first real LLM integration point**: a single enrichment call per Observe step that turns a `Scene` into a compact `SemanticScene` (page identity, supported intents, per-widget purpose labels), so a real LLM-driven `Planner` (a later milestone) and other prompts can consume a stable, low-token JSON summary instead of re-deriving page semantics from raw element geometry or a screenshot every step.
|
|
|
|
Because this is the first LLM call in the project, this design also sets the pattern (client shape, structured-output mechanism, failure handling) that later milestones (a real `Planner`, World Runtime's cross-step memory, Skill synthesis) are expected to reuse rather than inventing their own.
|
|
|
|
## Goals / Non-Goals
|
|
|
|
**Goals:**
|
|
- Given a `Scene`, produce a `SemanticScene` — `{page: str, intents: list[str], widgets: [{element_id, purpose}]}` — using exactly one LLM call.
|
|
- Guarantee the enrichment call never blocks or fails the Observe→Act loop: any failure (timeout, rate limit, malformed response, disabled config) degrades to "no semantic scene," and callers fall back to the raw `Scene`.
|
|
- Keep the LLM client abstraction narrow and swappable (mockable in tests without network access), scoped inside the new `semantic/` package rather than becoming a project-wide `llm/` dependency other capabilities must adopt.
|
|
- Produce output that is schema-valid JSON by construction (not "usually valid, occasionally needs a repair pass"), since downstream code will parse it directly into `SemanticScene`.
|
|
- Make enrichment optional and cheap enough to run on every Observe step without materially changing task latency or cost profile.
|
|
|
|
**Non-Goals:**
|
|
- No change to `scene-perception`'s `Scene` format or fusion logic — `SemanticScene` is a separate, additive artifact layered on top, never a replacement.
|
|
- No persistent cross-step semantic state, caching of `SemanticScene` across steps, or diffing between steps — that is Milestone 6 (World Runtime).
|
|
- No skill synthesis or action recommendation derived from intents/purposes — that is Milestone 7 (Skill).
|
|
- No change to `Planner`/`Executor` control flow itself — this change only makes a new artifact available; wiring a real LLM-driven `Planner` to consume `SemanticScene` is left to a later milestone (see Migration Plan).
|
|
- No multi-provider LLM abstraction (e.g. supporting both Anthropic and OpenAI behind a common interface) — a single concrete client is enough for this milestone; a provider-agnostic port can be extracted later if a second provider is actually needed (YAGNI).
|
|
|
|
## Decisions
|
|
|
|
### D1: New `semantic/` package, not folded into `perception/` or `runtime/`
|
|
`SemanticScene` is conceptually "one layer above Scene," but it has fundamentally different runtime characteristics: it makes a network call, it can fail/timeout, and it is optional. Mixing it into `perception/` (which is currently synchronous, local-only, and always-on) would force every `perception/` consumer to reason about network failure modes it doesn't otherwise have. A sibling `semantic/` package (`models.py`, `llm_client.py`, `enricher.py`, `prompts.py`) keeps `perception/`'s contract unchanged and makes the enrichment layer's optionality explicit at the package boundary.
|
|
**Alternative considered**: add an `enrich()` method directly on `Scene` or inside `scene_builder.py`. Rejected — it would make `perception/` depend on an LLM SDK and network config, contradicting `device-agent-runtime-foundation`'s stated dependency direction (`perception` must stay a local, deterministic pipeline; LLM-backed behavior is explicitly scoped to sit above it, closer to `runtime`).
|
|
|
|
### D2: `enrich_scene()` returns `SemanticScene | None`, never raises for expected failure modes
|
|
`enricher.enrich_scene(scene, *, context=None) -> SemanticScene | None` catches all expected LLM-client failure modes (timeout, rate limit, connection error, malformed/schema-invalid response, enrichment disabled by config) internally and returns `None` on any of them, logging the reason. Callers (tools, later a real `Planner`) treat `None` as "use the raw `Scene`," never as an exception to handle.
|
|
**Alternative considered**: raise a typed `EnrichmentUnavailableError` and require every caller to catch it. Rejected — experience from `runtime/executor.py`'s existing retry/backoff pattern shows try/except-per-call-site is exactly the kind of boilerplate that leads to a caller eventually forgetting to catch it, silently turning a "nice-to-have" feature into a hard dependency. A `None`-returning function makes the degrade path the *only* path for callers to write, not an opt-in one.
|
|
|
|
### D3: Structured output via schema-constrained response, not free-text parsing or prompt-based JSON coaxing
|
|
The enrichment call uses the Messages API's structured-output mechanism (`output_config: {format: {type: "json_schema", schema: {...}}}`, or the SDK's `messages.parse()` helper with a `SemanticScene`-shaped Pydantic model) so the response is schema-valid JSON by construction, rather than asking the model to "reply with JSON" in the prompt and parsing/repairing free text. This directly eliminates an entire class of degrade-path triggers (truncated/wrapped/commented JSON) that a prompt-only approach would otherwise have to detect and handle.
|
|
**Alternative considered**: frame enrichment as a tool call with `strict: true` tool-input validation instead of a direct structured message reply. Rejected for this milestone — a direct structured response is simpler (no tool-loop bookkeeping) for a single, non-interactive extraction call; the tool-call pattern is more valuable when the model needs to choose between actions, which does not apply here.
|
|
|
|
### D4: Default model is a small/fast tier (`claude-haiku-4-5`), overridable via config
|
|
Enrichment is a bounded, low-complexity extraction task (label a page, list a handful of intents, tag widgets already enumerated in the `Scene`) invoked once per Observe step, so latency and per-call cost dominate the choice over raw capability. `claude-haiku-4-5` is the default: cheapest/fastest tier that still supports schema-constrained structured output. The model is a config value, not hardcoded, so a later milestone can upgrade the default if enrichment quality proves insufficient in practice without a code change.
|
|
**Alternative considered**: default to the project's most capable tier (as a generic "always use the best model" policy would suggest). Rejected specifically for this call site — the enrichment task is simple extraction over an already-structured `Scene` (not open-ended reasoning), and this call sits in the hot path of every Observe step, where added latency directly slows the whole task loop. A stronger model remains available via config for callers who need it (e.g. a harder-to-classify page).
|
|
|
|
### D5: No extended-thinking / effort tuning on the enrichment call
|
|
The enrichment call omits `thinking` entirely and does not set `output_config.effort`. `claude-haiku-4-5` is a fast-extraction model; forcing extra reasoning depth on a bounded-output classification/labeling task increases latency and cost with no expected quality benefit here, and some effort/thinking parameter combinations are rejected outright on newer model tiers depending on model family. Keeping the request shape minimal (system prompt + schema + scene JSON) also maximizes what can be prompt-cached (see D6).
|
|
**Alternative considered**: enable adaptive thinking for robustness on ambiguous screens. Rejected for the default path — if a later milestone finds enrichment quality insufficient on complex/ambiguous screens, this is a config-level model/parameter change, not a structural one.
|
|
|
|
### D6: Prompt caching on the fixed system/schema prefix
|
|
The enrichment system prompt (instructions + JSON schema description) is static across every call within a task (and across tasks), while only the `Scene` JSON body varies per call. The client marks the system-prompt block with a `cache_control: {"type": "ephemeral"}` breakpoint so repeated per-step calls within a task reuse the cached prefix instead of paying full input-token price every step. This is a pure cost optimization with no behavior change; if the fixed prefix is ever short enough to fall under a given model's minimum cacheable-prefix length, caching simply has no effect (not an error) — the design must not depend on caching succeeding for correctness.
|
|
**Alternative considered**: no caching, accept repeated system-prompt cost. Rejected — enrichment is invoked once per Observe step, so a multi-step task repeats the identical instructions/schema text on every call; caching this prefix is a low-effort, no-risk win once the per-step call pattern exists.
|
|
|
|
### D7: New `describe_screen_semantic` tool wraps the existing tool, existing tool untouched
|
|
A new `tools/describe_screen_semantic.py` calls the existing `tools/describe_screen.py` to get a `Scene`, then calls `enrich_scene()`, and returns `{scene, semantic_scene}` (with `semantic_scene` possibly `None`). `tools/describe_screen.py` itself is not modified — existing callers (including the current `Planner` stub and any wiring in `runtime/executor.py`'s `default_tool_registry()`) keep working unchanged, and the new tool is additive to the registry.
|
|
**Alternative considered**: add an `enrich: bool = False` kwarg directly to `describe_screen()`. Rejected — it would make `tools/describe_screen.py` depend on the new `semantic/` package (and transitively on LLM client config) even for callers who never pass `enrich=True`, which cuts against `scene-perception`'s existing "no LLM dependency" boundary. A separate wrapper tool keeps the dependency opt-in at the import level, not just the call-site level.
|
|
|
|
## Risks / Trade-offs
|
|
|
|
- **[Risk] Enrichment latency adds to every Observe step even when unused by the caller** → Mitigation: enrichment is only invoked through the new `describe_screen_semantic` tool / explicit `enrich_scene()` call, never automatically inside the existing `describe_screen`; callers who don't need it pay zero extra latency.
|
|
- **[Risk] LLM cost scales with task step count (one call per Observe step)** → Mitigation: cheap default model tier (D4) + prompt caching of the fixed prefix (D6); config exposes a global enable/disable switch so cost-sensitive environments (CI, offline dev) can turn enrichment off entirely.
|
|
- **[Risk] Enrichment quality (wrong page label, missed intent, mislabeled widget purpose) is not directly testable against real model output in unit tests** → Mitigation: `llm_client` is injected/mockable in `enricher.py`, so unit tests exercise the parsing/degrade-path logic against canned responses; only a small, explicitly-marked integration test exercises the real API (matching the existing `apex-agent-mvp` pattern of skippable hardware/network integration tests).
|
|
- **[Risk] `element_id` values in `widgets[].element_id` could drift from the `Scene`'s actual element IDs if the model hallucinates or omits one** → Mitigation: `enricher.py` validates every returned `element_id` against the input `Scene`'s element ID set post-response; any widget referencing an unknown ID is dropped (not treated as a hard failure) rather than propagated as a dangling reference.
|
|
- **[Risk] Introducing the project's first external LLM dependency adds a new operational failure mode (network, auth, rate limits) that didn't exist before** → Mitigation: this is exactly why D2's `None`-returning contract and the mandatory fallback-to-raw-`Scene` path exist; the entire capability is designed so this failure mode degrades gracefully rather than being a new way for tasks to fail.
|
|
|
|
## Migration Plan
|
|
|
|
This is a purely additive change with no rollback complexity beyond removing the new package and config:
|
|
1. Add the `semantic/` package and `tools/describe_screen_semantic.py`; wire the new tool into `runtime/executor.py`'s `default_tool_registry()` under its own key (e.g. `"describe_screen_semantic"`), alongside (not replacing) the existing `describe_screen` entry.
|
|
2. Add the `anthropic` SDK dependency and `semantic*` to `pyproject.toml`'s package-find include list; add enrichment config (enabled/disabled, model name) with enrichment defaulting to **disabled** until a caller explicitly opts in, so applying this change never silently changes existing task behavior or cost.
|
|
3. No data migration, no changes to `storage/`'s timeline format in this change — if a later milestone wants `SemanticScene` persisted per step, that is a `task-memory` capability change to propose separately.
|
|
4. Rollback is simply not calling the new tool / leaving enrichment disabled in config; no existing capability needs to be reverted.
|
|
|
|
## Open Questions
|
|
|
|
- Should `SemanticScene` (when present) be recorded in `TaskContext`/the timeline alongside `Scene`, or is it purely a same-step, non-persisted artifact until World Runtime (Milestone 6) defines cross-step state? Leaning toward "not persisted in this change" to keep the boundary with Milestone 6 clean, but this affects whether `runtime/task.py` needs any change at all in this milestone.
|
|
- Should the default enrichment model become configurable per-task (e.g. a harder app might warrant a stronger tier) or is a single global default sufficient until real usage data exists? Leaning toward global-default-only for this milestone.
|
|
- Exact timeout budget for the enrichment call (e.g. 5s vs 10s) before triggering the degrade path — needs to be tuned against real latency data once implemented; not fixed in this design.
|
|
- Should intents be a fully open-vocabulary list (whatever the model says) or validated against a small controlled vocabulary to keep them stable/comparable across steps and pages? Leaning open-vocabulary for this milestone since no downstream consumer (yet) needs a fixed taxonomy; revisit if World/Skill milestones need one.
|