feat: checkpoint device agent runtime milestones
This commit is contained in:
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-06
|
||||
@@ -0,0 +1,65 @@
|
||||
## Context
|
||||
|
||||
The repository is currently empty (only tooling config exists: `pyproject.toml`, `.gitignore`, editor/agent config dirs, and this `openspec/` planning tree). We are building the first version of **Apex Agent**, an AI-native iPhone automation platform, from scratch. The defining constraint from the proposal is that the LLM must never be coupled to a specific automation framework (Appium/WDA) — it only ever sees semantic tools and a unified `Scene` model. Everything described here is new; there is no legacy system to interoperate with.
|
||||
|
||||
Primary stakeholder/consumer: an LLM (GPT/Claude/Qwen/DeepSeek) acting as the agent "brain," calling into this platform either through MCP tool calls or REST, to drive one or more physical iPhones running WebDriverAgent.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Ship a working end-to-end loop: Observe (screenshot+OCR+tree → Scene) → Think (LLM/Planner) → Act (tool call → Driver → WDA) → Observe again.
|
||||
- Keep the LLM-facing surface 100% driver-agnostic: no Appium/WDA/XCUIElement types ever appear in tool names, parameters, or the Scene model.
|
||||
- Make the `Driver` interface implementable by something other than WDA later (e.g. `AndroidDriver`) without touching `tools/`, `vision/`, `runtime/`, or `api/`.
|
||||
- Persist a per-task timeline (screenshot/ocr/tree/prompt/action/result) so a task can eventually be replayed or debugged.
|
||||
- Expose the capability layer as MCP tools (primary) and a thin REST API (secondary, useful for debugging/dashboards).
|
||||
|
||||
**Non-Goals:**
|
||||
- No script recorder, no visual flow/orchestration editor, no "keystroke wizard" style macro engine.
|
||||
- No second driver implementation (Android/browser/Windows) in this change — only the interface must allow it.
|
||||
- No production task queue (Redis/RabbitMQ) or horizontal scaling — synchronous, single-process execution is sufficient for the MVP.
|
||||
- No PostgreSQL/MinIO — SQLite + local filesystem is sufficient for the MVP; swapping storage later must not require changing the `runtime/` or `tools/` layers.
|
||||
- No icon-detection model beyond a basic/stubbed implementation (`icon_detector.py` may return "not found" or use simple template matching); full CV-based icon recognition is future work.
|
||||
|
||||
## Decisions
|
||||
|
||||
### D1: Capability Layer sits between Agent Runtime and Driver, and is the only thing tools/ calls
|
||||
The `tools/` package (screenshot, tap, swipe, input_text, launch_app, ui_tree, describe_screen) is a thin wrapper that calls the currently active `Driver` instance obtained from `device/` (device manager/registry), and nothing else. `runtime/` (Planner/Executor) only ever calls functions in `tools/`, never a `Driver` directly.
|
||||
- **Alternative considered**: Let the Agent Runtime call `Driver` directly and skip the `tools/` layer. Rejected — this would make MCP tool schemas and Driver method signatures the same thing, so any future driver-specific quirk (e.g. Android needing a different tap gesture) leaks into the LLM-facing tool contract.
|
||||
|
||||
### D2: `Driver` is a stateless abstract interface; `WDADriver` is the only concrete implementation for this change
|
||||
`Driver` (in `core/driver.py`) defines `connect/disconnect/screenshot/tap/swipe/input/launch/terminate/tree/home/lock/unlock`. `core/wda_driver.py` implements it using the Appium Python client (talking to WebDriverAgent over HTTP). The driver holds no business/task state — only the live connection handle to one physical device.
|
||||
- **Alternative considered**: Call WDA's HTTP API directly instead of going through Appium's Python client. Deferred, not rejected — Appium client is faster to get working for the MVP; a follow-up change can swap to raw WDA HTTP calls behind the same `Driver` interface if Appium proves too heavy, without affecting any other layer.
|
||||
|
||||
### D3: Scene is the single perception artifact the LLM ever consumes
|
||||
`vision/scene_builder.py` fuses a screenshot, the accessibility/UI tree, and an OCR pass into one `Scene` object: `{ screen: {width, height}, elements: [{id, type, text, bounds, confidence}] }`. `tools/describe_screen.py` and `tools/find_text.py`/`find_icon()`-style helpers operate only on `Scene`, never on raw XCUIElement XML or raw OCR boxes.
|
||||
- **Alternative considered**: Expose UI tree (XML) and OCR results as two separate tool calls and let the LLM merge them. Rejected — pushes fusion logic (dedup, coordinate reconciliation) onto the LLM, which is unreliable and burns context; fusing once in `scene_builder.py` is strictly cheaper and more consistent.
|
||||
|
||||
### D4: Planner produces a step plan; Executor is the only thing that calls tools and retries
|
||||
`runtime/planner.py` takes a natural-language goal + current `Scene` and returns a small ordered list of intended steps (e.g. "find search box", "tap it", "type query"). `runtime/executor.py` turns each planned step into concrete tool calls, owns retry/backoff/wait-for-element logic, and re-invokes the Planner (or asks the LLM again) when a step fails or the Scene doesn't match expectations.
|
||||
- **Alternative considered**: Let the LLM emit raw tool calls one at a time with no separate Planner ("ReAct"-style, no plan object). Rejected per the proposal's explicit design goal (P-level principle from the source discussion: "AI 不容易跑飞") — an explicit plan gives the Executor something to check progress against and reduces runaway/looping behavior; the Planner step is intentionally kept lightweight for the MVP (no long-horizon planning algorithm required).
|
||||
|
||||
### D5: Task Memory writes one (screenshot, Scene JSON, prompt, tool call, result) record per step to local disk, indexed by task id
|
||||
`storage/timeline.py` + `storage/artifact_store.py` write `tasks/history/<task_id>/NNN.png` and `NNN.json` (Scene + prompt + tool call + result) on every Executor step. SQLite holds task/device metadata (task id, device id, status, timestamps); images and per-step JSON stay on the filesystem, not in the DB.
|
||||
- **Alternative considered**: Store everything (including images) in SQLite as blobs. Rejected — bloats the DB, complicates the later MinIO migration mentioned in the proposal's deferred work, and gains nothing since images are always accessed by path, never queried.
|
||||
|
||||
### D6: MCP Server is the primary LLM-facing surface; REST is a secondary/debug surface over the same `tools/` functions
|
||||
`api/mcp.py` registers each `tools/` function as an MCP tool (name, description, JSON-schema params) with no translation layer beyond argument (de)serialization. `api/rest.py` exposes the same underlying functions via FastAPI routes (`GET /devices`, `POST /devices/{id}/tap`, etc.) mainly for manual testing/dashboards, not as the primary agent interface.
|
||||
- **Alternative considered**: Build REST first and put MCP behind/on top of it (MCP server calls the REST API internally). Rejected — adds an unnecessary network hop and serialization layer for the primary path; both surfaces calling `tools/` directly keeps latency low and keeps `tools/` as the true single source of truth.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **[Risk]** WDA/Appium connections are flaky (device sleep, WDA process crash, USB/Wi-Fi drop) → **Mitigation**: `Driver.connect()`/`Executor` retry logic treats connection loss as a recoverable error with bounded retries and surfaces a clear `device offline` state to the Device Manager rather than silently hanging.
|
||||
- **[Risk]** OCR + UI tree fusion in `scene_builder.py` may produce duplicate/conflicting elements (same visual button reported by both tree and OCR) → **Mitigation**: dedupe by bounding-box overlap (IoU threshold) in the MVP, prefer the UI-tree entry when both exist since it has ground-truth bounds; document this as a known heuristic, not a solved problem.
|
||||
- **[Risk]** Treating the Planner as "lightweight" could let the agent loop indefinitely on a stuck screen → **Mitigation**: Executor enforces a max-step and max-retry ceiling per task, at which point the task fails loudly (recorded in the timeline) instead of looping silently.
|
||||
- **[Trade-off]** Synchronous, single-process execution (no queue) means only as many concurrent tasks as the process can handle in-loop, and one slow/stuck task blocks its device — acceptable for MVP given the proposal explicitly defers queueing to a later change.
|
||||
- **[Trade-off]** SQLite + local filesystem storage doesn't survive multi-host deployment — acceptable for MVP (single machine, small number of physical devices); the proposal already earmarks Postgres/MinIO as deferred follow-up.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
Not applicable — this is the first change in a new, empty repository. There is no prior system state or data to migrate, and no rollback target other than "delete the new code."
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Which OCR engine ships as the MVP default (PaddleOCR, as suggested in the source discussion, vs. a lighter-weight/cloud alternative) — deferred to implementation time in `tasks.md`, since it doesn't change any interface in this design.
|
||||
- Whether the Planner should call the same LLM the caller is using, or be allowed a separate/cheaper model for step planning — left open; MVP can start by having the caller's LLM do both planning and per-step reasoning through the MCP tools, with `runtime/planner.py` as a thin pass-through it can later replace.
|
||||
- Exact MCP SDK/library choice for `api/mcp.py` — to be resolved during Sprint 4 implementation, doesn't affect the capability contracts defined here.
|
||||
@@ -0,0 +1,32 @@
|
||||
## Why
|
||||
|
||||
Traditional "device farm" (群控) tools work by having a human write a fixed automation script that the system blindly replays. That model breaks the moment a UI changes, and it caps the system at whatever the script author anticipated. This project inverts the model: an LLM observes the real screen state, decides the next action itself, and the system's only job is to expose stable, semantic capabilities (screenshot, tap, OCR, launch, ...) that the LLM can call. We are building **Apex Agent**, an AI-native iPhone Agent Platform (working name "IPA"), starting from an empty repository. WebDriverAgent (WDA) is treated as just one interchangeable driver behind a capability layer, not the platform itself — so Android/other drivers can be added later without touching the agent or tool layers.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Introduce a **Device Manager** that discovers, connects to, and tracks the lifecycle/state (idle/busy/offline/error) of physical iPhones.
|
||||
- Introduce a **driver-independent Capability Layer** (screenshot, tap, swipe, input, launch, terminate, tree, home, lock/unlock) implemented first via a stateless **WDA Driver** adapter (Appium Python Client / WDA HTTP), designed so a future `AndroidDriver` can implement the same interface.
|
||||
- Introduce a **Vision/Perception pipeline** that fuses screenshot + UI tree + OCR into a single unified **Scene** model (JSON: screen size + elements with type/text/bounds/confidence) — the only representation the AI ever sees; raw XCUIElement/XML types are never exposed.
|
||||
- Introduce an **Agent Runtime** (Planner + Executor + Memory/Context) that runs the Observe → Think → Act → Observe loop, decomposing a high-level goal (e.g. "open Taobao and search Mac mini") into tool calls, with retry/wait handling in the Executor so the LLM doesn't have to micromanage timing.
|
||||
- Introduce a **Task Memory / Timeline** store that persists each step's screenshot, OCR result, UI tree, prompt, tool call, and result to disk (per-task history), laying the groundwork for future replay.
|
||||
- Introduce an **MCP Tool Server** (and a thin REST API) that exposes all of the above as semantic, LLM-facing tools (`take_screenshot`, `tap`, `swipe`, `input_text`, `launch_app`, `find_text`, `find_icon`, `get_ui_tree`, `describe_screen`, `list_devices`, `device_status`, ...) so any MCP-compatible client (Claude Desktop, GPT function calling, etc.) can drive a real iPhone without ever knowing Appium/WDA/HTTP exist underneath.
|
||||
- Explicitly out of scope for this change (non-goals): scripted "keystroke wizard" style macros, a script recorder/IDE, a visual flow orchestrator, Android/browser/Windows drivers (interface must allow them later, but no second driver ships now), and a production task queue (Redis/RabbitMQ) — synchronous execution is enough for the MVP loop.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `device-management`: Device discovery/connection/state tracking (DeviceManager) plus the driver-independent device capability interface (connect/disconnect/screenshot/tap/swipe/input/launch/terminate/tree/home/lock/unlock) and its first concrete implementation, the stateless WDA driver.
|
||||
- `scene-perception`: Turns a raw screenshot + UI tree + OCR pass into the unified Scene JSON model that the agent and LLM consume instead of raw XML/XCUIElement data.
|
||||
- `agent-runtime`: Planner + Executor + Memory/Context implementing the Observe → Think → Act → Observe loop, including retry and wait handling around tool calls.
|
||||
- `task-memory`: Per-task timeline persistence of each step's screenshot, OCR, tree, prompt, tool call, and result to local storage (artifact store), forming the basis for future replay.
|
||||
- `mcp-tool-server`: MCP server (plus thin REST surface) exposing device, perception, and agent capabilities as stable, semantic tools for LLM function calling, with no Appium/WDA concepts leaking through.
|
||||
|
||||
### Modified Capabilities
|
||||
(none — this is the first change in an empty project)
|
||||
|
||||
## Impact
|
||||
|
||||
- **New code**: entire initial codebase — `core/` (device manager, driver interface, WDA driver, scene model, memory, models), `tools/` (screenshot, tap, swipe, input_text, launch_app, ui_tree, describe_screen), `vision/` (ocr, ui_parser, scene_builder, icon_detector), `runtime/` (planner, executor, context, task), `api/` (rest, mcp), `storage/` (timeline, artifact_store), `tests/`.
|
||||
- **Dependencies**: Python 3.14 (per existing `pyproject.toml`), Appium Python Client / WDA HTTP client, an OCR engine (e.g. PaddleOCR), FastAPI, an MCP server SDK, SQLite for MVP metadata storage, local filesystem for screenshots/artifacts.
|
||||
- **External systems**: requires a real (or simulator) iPhone reachable via WebDriverAgent/Appium; no cloud services required for the MVP.
|
||||
- **Follow-on work explicitly deferred**: Android/browser/Windows drivers, PostgreSQL migration, MinIO/object storage, Redis/RabbitMQ task queue, icon-detection model beyond a basic stub.
|
||||
@@ -0,0 +1,41 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Observe-Think-Act-Observe execution loop
|
||||
The system SHALL execute a task as a repeating loop: observe the current Scene, decide the next step, act via a capability tool call, then observe the resulting Scene again, continuing until the goal is met, a failure ceiling is hit, or the task is cancelled.
|
||||
|
||||
#### Scenario: Loop continues until goal is met
|
||||
- **WHEN** a task with a natural-language goal (e.g. "open Taobao and search Mac mini") is started
|
||||
- **THEN** the runtime repeats observe→think→act until the Planner/Executor determines the goal has been reached, then marks the task complete
|
||||
|
||||
#### Scenario: Loop stops after max steps
|
||||
- **WHEN** a task exceeds a configured maximum number of steps without reaching its goal
|
||||
- **THEN** the runtime stops the loop and marks the task as failed with a reason, instead of looping indefinitely
|
||||
|
||||
### Requirement: Planner produces a step plan from goal and current Scene
|
||||
The system SHALL provide a Planner that, given a goal and the current Scene, produces an ordered list of intended next steps (e.g. "find search box", "tap it", "type query").
|
||||
|
||||
#### Scenario: Planner emits steps for a new goal
|
||||
- **WHEN** the Planner is invoked with a goal and the current Scene at the start of a task
|
||||
- **THEN** it returns a non-empty ordered list of intended steps for the Executor to attempt
|
||||
|
||||
#### Scenario: Planner re-plans after unexpected Scene
|
||||
- **WHEN** the Executor reports that the Scene after an action does not match what the current step expected
|
||||
- **THEN** the Planner is invoked again with the updated Scene to produce a revised step (or remaining steps)
|
||||
|
||||
### Requirement: Executor performs retry and wait handling around tool calls
|
||||
The system SHALL provide an Executor that translates a planned step into one or more capability tool calls, and SHALL retry with backoff and/or wait-for-element behavior when a step's expected result is not immediately observed, up to a configured retry ceiling.
|
||||
|
||||
#### Scenario: Executor retries a transient failure
|
||||
- **WHEN** a tool call (e.g. `tap`) does not produce the expected Scene change on the first attempt
|
||||
- **THEN** the Executor retries the step up to a configured number of attempts before treating it as failed
|
||||
|
||||
#### Scenario: Executor gives up after retry ceiling
|
||||
- **WHEN** a step has failed for the configured maximum number of retries
|
||||
- **THEN** the Executor records the step as failed and surfaces this to the Planner/task result instead of retrying forever
|
||||
|
||||
### Requirement: Task context/memory available during a run
|
||||
The system SHALL maintain an in-run context (recent Scene history, executed steps, and their results) accessible to the Planner and Executor for the duration of a task, so re-planning decisions can reference what has already been tried.
|
||||
|
||||
#### Scenario: Re-planning uses prior step history
|
||||
- **WHEN** the Planner is re-invoked mid-task
|
||||
- **THEN** it has access to the steps already attempted in this task and their outcomes, not just the current Scene in isolation
|
||||
@@ -0,0 +1,52 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Device discovery and listing
|
||||
The system SHALL provide a Device Manager that can list all known iPhone devices and their current status (`idle`, `busy`, `offline`, `error`).
|
||||
|
||||
#### Scenario: Listing devices returns current status
|
||||
- **WHEN** a caller requests the device list
|
||||
- **THEN** the system returns each known device's id, status, and driver connection info (e.g. WDA port) without contacting the physical device for every field
|
||||
|
||||
#### Scenario: No devices connected
|
||||
- **WHEN** a caller requests the device list and no physical devices are reachable
|
||||
- **THEN** the system returns an empty list rather than raising an error
|
||||
|
||||
### Requirement: Device connect and disconnect lifecycle
|
||||
The system SHALL allow a caller to connect to and disconnect from a specific device by id, transitioning its tracked status accordingly.
|
||||
|
||||
#### Scenario: Successful connect
|
||||
- **WHEN** a caller connects to a device id that is currently `idle` and reachable
|
||||
- **THEN** the Device Manager marks the device `busy`, and it becomes usable for capability calls
|
||||
|
||||
#### Scenario: Connect to unreachable device
|
||||
- **WHEN** a caller connects to a device id that cannot be reached (WDA not responding)
|
||||
- **THEN** the Device Manager marks the device `offline` and returns an error to the caller instead of hanging indefinitely
|
||||
|
||||
#### Scenario: Disconnect releases the device
|
||||
- **WHEN** a caller disconnects from a device it previously connected to
|
||||
- **THEN** the Device Manager releases the underlying driver connection and marks the device `idle`
|
||||
|
||||
### Requirement: Driver-independent capability interface
|
||||
The system SHALL define a single `Driver` interface (`connect`, `disconnect`, `screenshot`, `tap`, `swipe`, `input`, `launch`, `terminate`, `tree`, `home`, `lock`, `unlock`) that any concrete driver implementation (e.g. WDA, and in future Android) must satisfy identically, so callers above the driver layer never depend on a specific automation framework.
|
||||
|
||||
#### Scenario: Capability call is dispatched through the interface
|
||||
- **WHEN** a higher layer (tools/) invokes a capability such as `tap(x, y)` on a connected device
|
||||
- **THEN** the call is routed through the `Driver` interface to the concrete driver instance for that device, with no framework-specific (e.g. Appium/WDA) types or errors surfacing to the caller
|
||||
|
||||
### Requirement: WDA driver implementation
|
||||
The system SHALL provide a concrete `WDADriver` implementing the `Driver` interface using WebDriverAgent (via Appium Python client), supporting at minimum: screenshot capture, tap, swipe, text input, app launch, app terminate, UI tree retrieval, home button, and device lock/unlock.
|
||||
|
||||
#### Scenario: Screenshot via WDA driver
|
||||
- **WHEN** `screenshot()` is called on a device backed by `WDADriver`
|
||||
- **THEN** the driver returns image bytes/path representing the current physical screen contents
|
||||
|
||||
#### Scenario: Launch app via WDA driver
|
||||
- **WHEN** `launch(bundle_id_or_name)` is called on a device backed by `WDADriver`
|
||||
- **THEN** the driver starts the requested app on the physical device and the call returns once the app process is confirmed running (or raises a clear error if launch fails)
|
||||
|
||||
### Requirement: Stateless driver
|
||||
Driver implementations SHALL NOT persist task-level or business state (e.g. current task id, plan progress); any such state SHALL be owned by the Agent Runtime / Task Memory layers, not the driver.
|
||||
|
||||
#### Scenario: Driver restart does not lose task progress
|
||||
- **WHEN** a driver connection is dropped and re-established mid-task
|
||||
- **THEN** the in-progress task's plan, step history, and timeline remain intact because they were never stored in the driver, only the live device connection needs to be re-established
|
||||
@@ -0,0 +1,30 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: MCP tools expose capability layer without driver leakage
|
||||
The system SHALL expose an MCP server whose tools (at minimum: `take_screenshot`, `tap`, `swipe`, `input_text`, `launch_app`, `find_text`, `find_icon`, `get_ui_tree`, `describe_screen`, `list_devices`, `device_status`) map directly to the capability/perception layer, with tool names, parameters, and return values containing no Appium/WDA/XCUIElement-specific concepts.
|
||||
|
||||
#### Scenario: MCP client calls a tool without framework knowledge
|
||||
- **WHEN** an MCP-compatible LLM client (e.g. Claude Desktop) calls the `tap` tool with coordinates
|
||||
- **THEN** the call succeeds by being routed through the capability layer to the underlying driver, and neither the tool schema nor its response exposes Appium/WDA-specific types or errors
|
||||
|
||||
#### Scenario: Tool list is stable across driver changes
|
||||
- **WHEN** the underlying driver for a device changes (e.g. a future non-WDA driver is used) while the MCP tool set is unchanged
|
||||
- **THEN** existing MCP tool calls continue to work without any change to tool names or parameter schemas
|
||||
|
||||
### Requirement: MCP tool errors are semantic, not framework-specific
|
||||
The system SHALL translate driver/framework-level errors (e.g. WDA connection errors, element-not-found) into clear, semantic MCP tool error responses (e.g. "device offline", "element not found") rather than passing raw framework exceptions through to the LLM.
|
||||
|
||||
#### Scenario: Device offline surfaces a clear error
|
||||
- **WHEN** an MCP tool call targets a device that is currently offline
|
||||
- **THEN** the tool call returns a semantic error indicating the device is unavailable, not a raw connection-refused/stack-trace style error
|
||||
|
||||
### Requirement: REST API mirrors the same capability functions
|
||||
The system SHALL provide a REST API (`GET /devices`, `POST /devices/{id}/tap`, `POST /devices/{id}/screenshot`, `POST /devices/{id}/launch`, `POST /agent/task`, `GET /task/{id}`) that calls the same underlying capability functions as the MCP tools, for manual testing and dashboard use.
|
||||
|
||||
#### Scenario: REST and MCP produce consistent results
|
||||
- **WHEN** the same capability (e.g. `screenshot`) is invoked once via the REST API and once via the corresponding MCP tool for the same device
|
||||
- **THEN** both calls exercise the same underlying capability function and produce equivalent results
|
||||
|
||||
#### Scenario: Starting a task via REST
|
||||
- **WHEN** a caller POSTs a goal to `/agent/task`
|
||||
- **THEN** the system creates a new task, returns its task id, and the task becomes queryable via `GET /task/{id}`
|
||||
@@ -0,0 +1,37 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Unified Scene generation
|
||||
The system SHALL generate a single `Scene` object for the current screen by fusing a screenshot, the UI accessibility tree, and an OCR pass, so that no caller above the perception layer ever needs to read raw UI-tree XML or raw OCR output directly.
|
||||
|
||||
#### Scenario: Scene built from a live screen
|
||||
- **WHEN** the perception layer is asked to describe the current screen of a connected device
|
||||
- **THEN** it captures a screenshot, retrieves the UI tree, runs OCR, and returns one `Scene` object containing `screen` (width, height) and a list of `elements`, each with `id`, `type`, `text`, `bounds`, and `confidence`
|
||||
|
||||
#### Scenario: OCR-only element when tree has no match
|
||||
- **WHEN** OCR detects a text region that has no corresponding node in the UI tree
|
||||
- **THEN** the Scene SHALL still include that text as an element (type inferred as `text` or `unknown`), rather than silently dropping it
|
||||
|
||||
### Requirement: Duplicate element reconciliation
|
||||
The system SHALL reconcile elements that are reported by both the UI tree and OCR for the same visual region into a single Scene element, preferring UI-tree-provided bounds/type when both sources overlap significantly.
|
||||
|
||||
#### Scenario: Button text detected by both tree and OCR
|
||||
- **WHEN** a button's label is present both as a UI tree node and as an OCR text box with substantially overlapping bounds
|
||||
- **THEN** the Scene contains one element for that button, using the UI tree's type and bounds, with the OCR text merged in if the tree node lacked a text value
|
||||
|
||||
### Requirement: Semantic lookup helpers over Scene
|
||||
The system SHALL provide `find_text` and `find_icon` helpers that search the current `Scene` for a matching element and return its coordinates, without requiring the caller to parse the Scene JSON manually.
|
||||
|
||||
#### Scenario: Find text present on screen
|
||||
- **WHEN** a caller invokes `find_text("搜索")` while the current Scene contains an element with that text
|
||||
- **THEN** the system returns the tappable coordinates (center point or bounds) of the matching element
|
||||
|
||||
#### Scenario: Find text not present on screen
|
||||
- **WHEN** a caller invokes `find_text(...)` for a string that has no match in the current Scene
|
||||
- **THEN** the system returns a clear "not found" result rather than raising an unhandled exception
|
||||
|
||||
### Requirement: describe_screen returns Scene only
|
||||
The `describe_screen` tool SHALL return the Scene model (or a summarized natural-language rendering of it) and SHALL NOT return raw XCUIElement type names, raw XML, or driver-specific identifiers.
|
||||
|
||||
#### Scenario: describe_screen output is driver-agnostic
|
||||
- **WHEN** `describe_screen()` is called against a device backed by the WDA driver
|
||||
- **THEN** the returned content contains only Scene-model fields (screen size, elements with type/text/bounds) and no WDA/Appium/XCUIElement-specific terms
|
||||
@@ -0,0 +1,30 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Per-step timeline persistence
|
||||
The system SHALL persist a record for every step of a task, containing the screenshot, the Scene JSON, the prompt/goal context, the tool call issued, and its result, written to local storage under a task-specific path.
|
||||
|
||||
#### Scenario: Step record written after each action
|
||||
- **WHEN** the Executor completes a step (successful or failed)
|
||||
- **THEN** the system writes that step's screenshot and a JSON record (Scene, prompt, tool call, result) to the task's timeline directory before proceeding to the next step
|
||||
|
||||
#### Scenario: Timeline survives process restart
|
||||
- **WHEN** the process running a task restarts after a step has already been persisted
|
||||
- **THEN** the previously persisted steps for that task remain readable from disk
|
||||
|
||||
### Requirement: Task metadata storage
|
||||
The system SHALL store task-level metadata (task id, target device id, status, start/end timestamps) in a local database (SQLite for this change), separate from the per-step image/JSON artifacts on the filesystem.
|
||||
|
||||
#### Scenario: Task status queryable during execution
|
||||
- **WHEN** a caller queries a task by id while it is still running
|
||||
- **THEN** the system returns its current status (e.g. `running`) and metadata without needing to scan the filesystem timeline
|
||||
|
||||
#### Scenario: Task status reflects completion
|
||||
- **WHEN** a task finishes (successfully or with failure)
|
||||
- **THEN** its stored metadata status is updated accordingly and its end timestamp is recorded
|
||||
|
||||
### Requirement: Timeline retrievable for future replay
|
||||
The system SHALL store each task's steps in a way that preserves their order and completeness, so that a future replay feature can reconstruct the full sequence of screenshots/Scenes/actions for a task without additional data collection.
|
||||
|
||||
#### Scenario: Steps retrievable in order
|
||||
- **WHEN** a caller requests the full timeline for a completed task
|
||||
- **THEN** the system returns all persisted steps in the order they occurred, each with its screenshot and JSON record
|
||||
Reference in New Issue
Block a user