29 lines
4.6 KiB
Markdown
29 lines
4.6 KiB
Markdown
## Why
|
|
|
|
`scene-perception` (from `apex-agent-mvp`, code-complete but unapplied) fuses a screenshot, the device's UI tree, and OCR output into a `Scene`: screen dimensions plus a flat list of typed elements (bounds, text, confidence, source). That is still a *structural* description of the screen — it tells the LLM "there is a button-shaped element at (x, y) with text 'Send'," not "this is the WeChat chat screen and tapping that button sends the message." Every planning step, the LLM has to re-derive page identity and intent from raw element geometry, which burns context on repeated low-level reasoning and is brittle to layout drift (the same page re-derived slightly differently step to step). This change adds a **Semantic Runtime**: a single additional LLM enrichment call that turns a `Scene` into a `SemanticScene` — page identity, a short list of plain-language supported intents, and a purpose label per widget — so most subsequent prompts can rely on a compact, stable JSON summary instead of re-parsing raw geometry or screenshots. This is Milestone 5 (Semantic) of the device-agnostic runtime roadmap established by `device-agent-runtime-foundation`, sitting directly on top of the existing Perception milestone's `Scene` output.
|
|
|
|
## What Changes
|
|
|
|
- Add a new `semantic/` package that consumes an existing `Scene` (from `perception/`, per the `scene-perception` capability) and produces a `SemanticScene`: `{page: str, intents: list[str], widgets: [{element_id: str, purpose: str}]}`.
|
|
- Introduce the **first real LLM integration point** in the codebase: a narrow, swappable LLM client abstraction used only by `semantic/` to make one structured-output call per enrichment (model, prompt, and structured-schema details are an implementation decision in `design.md`, not part of this proposal's contract).
|
|
- Define a mandatory **degrade path**: if the enrichment LLM call is unavailable, times out, or fails for any reason, semantic enrichment is skipped and callers fall back to using the raw `Scene` directly — enrichment failure must never block or fail the Observe→Act loop.
|
|
- Add a new tool-level entry point (e.g. `describe_screen_semantic` or an `enrich` flag on the existing describe-screen flow) so `runtime/` and `api/` callers can opt into the enriched view without every existing caller of `describe_screen` needing to change.
|
|
- Add configuration to enable/disable semantic enrichment globally (so it can be turned off entirely in environments without LLM access, e.g. tests or offline development) without touching `scene-perception`.
|
|
- **BREAKING**: none. This is a purely additive layer; nothing in `scene-perception`, `agent-runtime`, or any other existing capability changes shape or behavior.
|
|
|
|
## Capabilities
|
|
|
|
### New Capabilities
|
|
- `semantic-scene`: Builds a `SemanticScene` (page identity, supported intents, per-widget purpose labels) from an existing `Scene` via one additional LLM call, with a defined skip/fallback degrade path when that call is unavailable or fails, so enrichment is strictly additive and non-blocking to the Observe→Act loop.
|
|
|
|
### Modified Capabilities
|
|
(none — `scene-perception`'s `Scene` format and requirements from `apex-agent-mvp` are read-only input to this change and are not modified; `agent-runtime`'s Planner/Executor loop shape is not changed by this proposal, only optionally consumed by it, see `design.md` for how a future milestone might wire it in.)
|
|
|
|
## Impact
|
|
|
|
- **New package**: `semantic/` — `models.py` (`SemanticScene`, `SemanticWidget` dataclasses), `llm_client.py` (narrow LLM client wrapper + typed result), `enricher.py` (`enrich_scene(scene, ...) -> SemanticScene | None`), `prompts.py` (the enrichment system prompt / JSON schema description).
|
|
- **New tool**: `tools/describe_screen_semantic.py` (or equivalent), wrapping the existing `tools/describe_screen.py` output with an enrichment pass and degrade path, following the same `manager`/`device_id` kwarg pattern as other tools.
|
|
- **Config**: `pyproject.toml` gains an `anthropic` SDK dependency and a `semantic*` entry in `[tool.setuptools.packages.find].include`; a new settings surface (e.g. env var or config value) to enable/disable enrichment and select the model.
|
|
- **No change** to `core/models.py`'s `Scene`/`SceneElement`, `vision`/`perception`'s fusion pipeline, or any existing tool signatures — all existing callers of `describe_screen` keep working unmodified.
|
|
- **Out of scope**: no persistent cross-step semantic state or caching across steps (Milestone 6, World Runtime); no skill synthesis from semantic labels (Milestone 7); no change to `skill-catalog-subscription` or `web-console`.
|