agentic-mobile-control/openspec/changes/apex-agent-mvp/specs/scene-perception/spec.md

ADDED Requirements

Requirement: Unified Scene generation

The system SHALL generate a single Scene object for the current screen by fusing a screenshot, the UI accessibility tree, and an OCR pass, so that no caller above the perception layer ever needs to read raw UI-tree XML or raw OCR output directly.

Scenario: Scene built from a live screen

• WHEN the perception layer is asked to describe the current screen of a connected device
• THEN it captures a screenshot, retrieves the UI tree, runs OCR, and returns one Scene object containing screen (width, height) and a list of elements, each with id, type, text, bounds, and confidence

Scenario: OCR-only element when tree has no match

• WHEN OCR detects a text region that has no corresponding node in the UI tree
• THEN the Scene SHALL still include that text as an element (type inferred as text or unknown), rather than silently dropping it

Requirement: Duplicate element reconciliation

The system SHALL reconcile elements that are reported by both the UI tree and OCR for the same visual region into a single Scene element, preferring UI-tree-provided bounds/type when both sources overlap significantly.

Scenario: Button text detected by both tree and OCR

• WHEN a button's label is present both as a UI tree node and as an OCR text box with substantially overlapping bounds
• THEN the Scene contains one element for that button, using the UI tree's type and bounds, with the OCR text merged in if the tree node lacked a text value

Requirement: Semantic lookup helpers over Scene

The system SHALL provide find_text and find_icon helpers that search the current Scene for a matching element and return its coordinates, without requiring the caller to parse the Scene JSON manually.

Scenario: Find text present on screen

• WHEN a caller invokes find_text("搜索") while the current Scene contains an element with that text
• THEN the system returns the tappable coordinates (center point or bounds) of the matching element

Scenario: Find text not present on screen

• WHEN a caller invokes find_text(...) for a string that has no match in the current Scene
• THEN the system returns a clear "not found" result rather than raising an unhandled exception

Requirement: describe_screen returns Scene only

The describe_screen tool SHALL return the Scene model (or a summarized natural-language rendering of it) and SHALL NOT return raw XCUIElement type names, raw XML, or driver-specific identifiers.

Scenario: describe_screen output is driver-agnostic

• WHEN describe_screen() is called against a device backed by the WDA driver
• THEN the returned content contains only Scene-model fields (screen size, elements with type/text/bounds) and no WDA/Appium/XCUIElement-specific terms