feat: checkpoint device agent runtime milestones

2026-07-06 17:24:03 +08:00
parent 2d4251e98e
commit 5658735bca
153 changed files with 8060 additions and 65 deletions
@@ -0,0 +1,41 @@
+## ADDED Requirements
+
+### Requirement: Observe-Think-Act-Observe execution loop
+The system SHALL execute a task as a repeating loop: observe the current Scene, decide the next step, act via a capability tool call, then observe the resulting Scene again, continuing until the goal is met, a failure ceiling is hit, or the task is cancelled.
+
+#### Scenario: Loop continues until goal is met
+- **WHEN** a task with a natural-language goal (e.g. "open Taobao and search Mac mini") is started
+- **THEN** the runtime repeats observe→think→act until the Planner/Executor determines the goal has been reached, then marks the task complete
+
+#### Scenario: Loop stops after max steps
+- **WHEN** a task exceeds a configured maximum number of steps without reaching its goal
+- **THEN** the runtime stops the loop and marks the task as failed with a reason, instead of looping indefinitely
+
+### Requirement: Planner produces a step plan from goal and current Scene
+The system SHALL provide a Planner that, given a goal and the current Scene, produces an ordered list of intended next steps (e.g. "find search box", "tap it", "type query").
+
+#### Scenario: Planner emits steps for a new goal
+- **WHEN** the Planner is invoked with a goal and the current Scene at the start of a task
+- **THEN** it returns a non-empty ordered list of intended steps for the Executor to attempt
+
+#### Scenario: Planner re-plans after unexpected Scene
+- **WHEN** the Executor reports that the Scene after an action does not match what the current step expected
+- **THEN** the Planner is invoked again with the updated Scene to produce a revised step (or remaining steps)
+
+### Requirement: Executor performs retry and wait handling around tool calls
+The system SHALL provide an Executor that translates a planned step into one or more capability tool calls, and SHALL retry with backoff and/or wait-for-element behavior when a step's expected result is not immediately observed, up to a configured retry ceiling.
+
+#### Scenario: Executor retries a transient failure
+- **WHEN** a tool call (e.g. `tap`) does not produce the expected Scene change on the first attempt
+- **THEN** the Executor retries the step up to a configured number of attempts before treating it as failed
+
+#### Scenario: Executor gives up after retry ceiling
+- **WHEN** a step has failed for the configured maximum number of retries
+- **THEN** the Executor records the step as failed and surfaces this to the Planner/task result instead of retrying forever
+
+### Requirement: Task context/memory available during a run
+The system SHALL maintain an in-run context (recent Scene history, executed steps, and their results) accessible to the Planner and Executor for the duration of a task, so re-planning decisions can reference what has already been tried.
+
+#### Scenario: Re-planning uses prior step history
+- **WHEN** the Planner is re-invoked mid-task
+- **THEN** it has access to the steps already attempted in this task and their outcomes, not just the current Scene in isolation
@@ -0,0 +1,52 @@
+## ADDED Requirements
+
+### Requirement: Device discovery and listing
+The system SHALL provide a Device Manager that can list all known iPhone devices and their current status (`idle`, `busy`, `offline`, `error`).
+
+#### Scenario: Listing devices returns current status
+- **WHEN** a caller requests the device list
+- **THEN** the system returns each known device's id, status, and driver connection info (e.g. WDA port) without contacting the physical device for every field
+
+#### Scenario: No devices connected
+- **WHEN** a caller requests the device list and no physical devices are reachable
+- **THEN** the system returns an empty list rather than raising an error
+
+### Requirement: Device connect and disconnect lifecycle
+The system SHALL allow a caller to connect to and disconnect from a specific device by id, transitioning its tracked status accordingly.
+
+#### Scenario: Successful connect
+- **WHEN** a caller connects to a device id that is currently `idle` and reachable
+- **THEN** the Device Manager marks the device `busy`, and it becomes usable for capability calls
+
+#### Scenario: Connect to unreachable device
+- **WHEN** a caller connects to a device id that cannot be reached (WDA not responding)
+- **THEN** the Device Manager marks the device `offline` and returns an error to the caller instead of hanging indefinitely
+
+#### Scenario: Disconnect releases the device
+- **WHEN** a caller disconnects from a device it previously connected to
+- **THEN** the Device Manager releases the underlying driver connection and marks the device `idle`
+
+### Requirement: Driver-independent capability interface
+The system SHALL define a single `Driver` interface (`connect`, `disconnect`, `screenshot`, `tap`, `swipe`, `input`, `launch`, `terminate`, `tree`, `home`, `lock`, `unlock`) that any concrete driver implementation (e.g. WDA, and in future Android) must satisfy identically, so callers above the driver layer never depend on a specific automation framework.
+
+#### Scenario: Capability call is dispatched through the interface
+- **WHEN** a higher layer (tools/) invokes a capability such as `tap(x, y)` on a connected device
+- **THEN** the call is routed through the `Driver` interface to the concrete driver instance for that device, with no framework-specific (e.g. Appium/WDA) types or errors surfacing to the caller
+
+### Requirement: WDA driver implementation
+The system SHALL provide a concrete `WDADriver` implementing the `Driver` interface using WebDriverAgent (via Appium Python client), supporting at minimum: screenshot capture, tap, swipe, text input, app launch, app terminate, UI tree retrieval, home button, and device lock/unlock.
+
+#### Scenario: Screenshot via WDA driver
+- **WHEN** `screenshot()` is called on a device backed by `WDADriver`
+- **THEN** the driver returns image bytes/path representing the current physical screen contents
+
+#### Scenario: Launch app via WDA driver
+- **WHEN** `launch(bundle_id_or_name)` is called on a device backed by `WDADriver`
+- **THEN** the driver starts the requested app on the physical device and the call returns once the app process is confirmed running (or raises a clear error if launch fails)
+
+### Requirement: Stateless driver
+Driver implementations SHALL NOT persist task-level or business state (e.g. current task id, plan progress); any such state SHALL be owned by the Agent Runtime / Task Memory layers, not the driver.
+
+#### Scenario: Driver restart does not lose task progress
+- **WHEN** a driver connection is dropped and re-established mid-task
+- **THEN** the in-progress task's plan, step history, and timeline remain intact because they were never stored in the driver, only the live device connection needs to be re-established
@@ -0,0 +1,30 @@
+## ADDED Requirements
+
+### Requirement: MCP tools expose capability layer without driver leakage
+The system SHALL expose an MCP server whose tools (at minimum: `take_screenshot`, `tap`, `swipe`, `input_text`, `launch_app`, `find_text`, `find_icon`, `get_ui_tree`, `describe_screen`, `list_devices`, `device_status`) map directly to the capability/perception layer, with tool names, parameters, and return values containing no Appium/WDA/XCUIElement-specific concepts.
+
+#### Scenario: MCP client calls a tool without framework knowledge
+- **WHEN** an MCP-compatible LLM client (e.g. Claude Desktop) calls the `tap` tool with coordinates
+- **THEN** the call succeeds by being routed through the capability layer to the underlying driver, and neither the tool schema nor its response exposes Appium/WDA-specific types or errors
+
+#### Scenario: Tool list is stable across driver changes
+- **WHEN** the underlying driver for a device changes (e.g. a future non-WDA driver is used) while the MCP tool set is unchanged
+- **THEN** existing MCP tool calls continue to work without any change to tool names or parameter schemas
+
+### Requirement: MCP tool errors are semantic, not framework-specific
+The system SHALL translate driver/framework-level errors (e.g. WDA connection errors, element-not-found) into clear, semantic MCP tool error responses (e.g. "device offline", "element not found") rather than passing raw framework exceptions through to the LLM.
+
+#### Scenario: Device offline surfaces a clear error
+- **WHEN** an MCP tool call targets a device that is currently offline
+- **THEN** the tool call returns a semantic error indicating the device is unavailable, not a raw connection-refused/stack-trace style error
+
+### Requirement: REST API mirrors the same capability functions
+The system SHALL provide a REST API (`GET /devices`, `POST /devices/{id}/tap`, `POST /devices/{id}/screenshot`, `POST /devices/{id}/launch`, `POST /agent/task`, `GET /task/{id}`) that calls the same underlying capability functions as the MCP tools, for manual testing and dashboard use.
+
+#### Scenario: REST and MCP produce consistent results
+- **WHEN** the same capability (e.g. `screenshot`) is invoked once via the REST API and once via the corresponding MCP tool for the same device
+- **THEN** both calls exercise the same underlying capability function and produce equivalent results
+
+#### Scenario: Starting a task via REST
+- **WHEN** a caller POSTs a goal to `/agent/task`
+- **THEN** the system creates a new task, returns its task id, and the task becomes queryable via `GET /task/{id}`
@@ -0,0 +1,37 @@
+## ADDED Requirements
+
+### Requirement: Unified Scene generation
+The system SHALL generate a single `Scene` object for the current screen by fusing a screenshot, the UI accessibility tree, and an OCR pass, so that no caller above the perception layer ever needs to read raw UI-tree XML or raw OCR output directly.
+
+#### Scenario: Scene built from a live screen
+- **WHEN** the perception layer is asked to describe the current screen of a connected device
+- **THEN** it captures a screenshot, retrieves the UI tree, runs OCR, and returns one `Scene` object containing `screen` (width, height) and a list of `elements`, each with `id`, `type`, `text`, `bounds`, and `confidence`
+
+#### Scenario: OCR-only element when tree has no match
+- **WHEN** OCR detects a text region that has no corresponding node in the UI tree
+- **THEN** the Scene SHALL still include that text as an element (type inferred as `text` or `unknown`), rather than silently dropping it
+
+### Requirement: Duplicate element reconciliation
+The system SHALL reconcile elements that are reported by both the UI tree and OCR for the same visual region into a single Scene element, preferring UI-tree-provided bounds/type when both sources overlap significantly.
+
+#### Scenario: Button text detected by both tree and OCR
+- **WHEN** a button's label is present both as a UI tree node and as an OCR text box with substantially overlapping bounds
+- **THEN** the Scene contains one element for that button, using the UI tree's type and bounds, with the OCR text merged in if the tree node lacked a text value
+
+### Requirement: Semantic lookup helpers over Scene
+The system SHALL provide `find_text` and `find_icon` helpers that search the current `Scene` for a matching element and return its coordinates, without requiring the caller to parse the Scene JSON manually.
+
+#### Scenario: Find text present on screen
+- **WHEN** a caller invokes `find_text("搜索")` while the current Scene contains an element with that text
+- **THEN** the system returns the tappable coordinates (center point or bounds) of the matching element
+
+#### Scenario: Find text not present on screen
+- **WHEN** a caller invokes `find_text(...)` for a string that has no match in the current Scene
+- **THEN** the system returns a clear "not found" result rather than raising an unhandled exception
+
+### Requirement: describe_screen returns Scene only
+The `describe_screen` tool SHALL return the Scene model (or a summarized natural-language rendering of it) and SHALL NOT return raw XCUIElement type names, raw XML, or driver-specific identifiers.
+
+#### Scenario: describe_screen output is driver-agnostic
+- **WHEN** `describe_screen()` is called against a device backed by the WDA driver
+- **THEN** the returned content contains only Scene-model fields (screen size, elements with type/text/bounds) and no WDA/Appium/XCUIElement-specific terms
@@ -0,0 +1,30 @@
+## ADDED Requirements
+
+### Requirement: Per-step timeline persistence
+The system SHALL persist a record for every step of a task, containing the screenshot, the Scene JSON, the prompt/goal context, the tool call issued, and its result, written to local storage under a task-specific path.
+
+#### Scenario: Step record written after each action
+- **WHEN** the Executor completes a step (successful or failed)
+- **THEN** the system writes that step's screenshot and a JSON record (Scene, prompt, tool call, result) to the task's timeline directory before proceeding to the next step
+
+#### Scenario: Timeline survives process restart
+- **WHEN** the process running a task restarts after a step has already been persisted
+- **THEN** the previously persisted steps for that task remain readable from disk
+
+### Requirement: Task metadata storage
+The system SHALL store task-level metadata (task id, target device id, status, start/end timestamps) in a local database (SQLite for this change), separate from the per-step image/JSON artifacts on the filesystem.
+
+#### Scenario: Task status queryable during execution
+- **WHEN** a caller queries a task by id while it is still running
+- **THEN** the system returns its current status (e.g. `running`) and metadata without needing to scan the filesystem timeline
+
+#### Scenario: Task status reflects completion
+- **WHEN** a task finishes (successfully or with failure)
+- **THEN** its stored metadata status is updated accordingly and its end timestamp is recorded
+
+### Requirement: Timeline retrievable for future replay
+The system SHALL store each task's steps in a way that preserves their order and completeness, so that a future replay feature can reconstruct the full sequence of screenshots/Scenes/actions for a task without additional data collection.
+
+#### Scenario: Steps retrievable in order
+- **WHEN** a caller requests the full timeline for a completed task
+- **THEN** the system returns all persisted steps in the order they occurred, each with its screenshot and JSON record