38 lines
2.8 KiB
Markdown
38 lines
2.8 KiB
Markdown
## ADDED Requirements
|
|
|
|
### Requirement: Unified Scene generation
|
|
The system SHALL generate a single `Scene` object for the current screen by fusing a screenshot, the UI accessibility tree, and an OCR pass, so that no caller above the perception layer ever needs to read raw UI-tree XML or raw OCR output directly.
|
|
|
|
#### Scenario: Scene built from a live screen
|
|
- **WHEN** the perception layer is asked to describe the current screen of a connected device
|
|
- **THEN** it captures a screenshot, retrieves the UI tree, runs OCR, and returns one `Scene` object containing `screen` (width, height) and a list of `elements`, each with `id`, `type`, `text`, `bounds`, and `confidence`
|
|
|
|
#### Scenario: OCR-only element when tree has no match
|
|
- **WHEN** OCR detects a text region that has no corresponding node in the UI tree
|
|
- **THEN** the Scene SHALL still include that text as an element (type inferred as `text` or `unknown`), rather than silently dropping it
|
|
|
|
### Requirement: Duplicate element reconciliation
|
|
The system SHALL reconcile elements that are reported by both the UI tree and OCR for the same visual region into a single Scene element, preferring UI-tree-provided bounds/type when both sources overlap significantly.
|
|
|
|
#### Scenario: Button text detected by both tree and OCR
|
|
- **WHEN** a button's label is present both as a UI tree node and as an OCR text box with substantially overlapping bounds
|
|
- **THEN** the Scene contains one element for that button, using the UI tree's type and bounds, with the OCR text merged in if the tree node lacked a text value
|
|
|
|
### Requirement: Semantic lookup helpers over Scene
|
|
The system SHALL provide `find_text` and `find_icon` helpers that search the current `Scene` for a matching element and return its coordinates, without requiring the caller to parse the Scene JSON manually.
|
|
|
|
#### Scenario: Find text present on screen
|
|
- **WHEN** a caller invokes `find_text("搜索")` while the current Scene contains an element with that text
|
|
- **THEN** the system returns the tappable coordinates (center point or bounds) of the matching element
|
|
|
|
#### Scenario: Find text not present on screen
|
|
- **WHEN** a caller invokes `find_text(...)` for a string that has no match in the current Scene
|
|
- **THEN** the system returns a clear "not found" result rather than raising an unhandled exception
|
|
|
|
### Requirement: describe_screen returns Scene only
|
|
The `describe_screen` tool SHALL return the Scene model (or a summarized natural-language rendering of it) and SHALL NOT return raw XCUIElement type names, raw XML, or driver-specific identifiers.
|
|
|
|
#### Scenario: describe_screen output is driver-agnostic
|
|
- **WHEN** `describe_screen()` is called against a device backed by the WDA driver
|
|
- **THEN** the returned content contains only Scene-model fields (screen size, elements with type/text/bounds) and no WDA/Appium/XCUIElement-specific terms
|