agentic-mobile-control/openspec/changes/apex-agent-mvp/tasks.md at 557c8a25baf83daf4457ec00987d5ef4ce4554b1

Files

T

2026-07-06 13:22:15 +08:00

1. Project scaffolding

1.1 Create the package layout: core/, tools/, vision/, runtime/, api/, storage/, tests/ (each with __init__.py)
1.2 Add MVP dependencies to pyproject.toml (Appium Python client, an OCR engine e.g. PaddleOCR, FastAPI, an MCP server SDK, SQLite driver, test framework)
1.3 Add core/models.py with shared data classes: Device, Scene, SceneElement, Task, Step
1.4 Set up a basic test runner config and a smoke test that imports every new package

2.1 Define the abstract Driver interface in core/driver.py (connect, disconnect, screenshot, tap, swipe, input, launch, terminate, tree, home, lock, unlock) with no stored business state
2.2 Implement core/device_manager.py: device discovery/registration, status tracking (idle/busy/offline/error), list_devices(), connect(), disconnect(), status()
2.3 Implement core/wda_driver.py (WDADriver) using the Appium Python client against WebDriverAgent, covering all Driver methods
2.4 Add connection-loss handling: mark device offline/error and bound retries on connect() instead of hanging
2.5 Write unit tests for DeviceManager state transitions using a fake/mock Driver
2.6 Write an integration test (skippable without hardware) that connects to a real/simulated device via WDADriver and takes a screenshot

3.1 Implement vision/ocr.py: run OCR over a screenshot and return text boxes with bounds/confidence
3.2 Implement vision/ui_parser.py: parse the raw UI tree returned by Driver.tree() into a normalized element list (type, text, bounds)
3.3 Implement vision/scene_builder.py: fuse OCR output + parsed UI tree into a single Scene, deduping overlapping elements by bounding-box IoU (prefer UI-tree bounds/type on overlap)
3.4 Implement vision/icon_detector.py as a minimal stub (template match or "not found") satisfying the find_icon contract
3.5 Implement tools/screenshot.py, tools/tap.py, tools/swipe.py, tools/input_text.py, tools/launch_app.py, tools/ui_tree.py as thin wrappers over the active device's Driver
3.6 Implement tools/describe_screen.py (returns the fused Scene) and find_text/find_icon helpers that search a Scene and return coordinates or "not found"
3.7 Write unit tests for scene_builder dedup logic using fixture screenshots/tree/OCR data
3.8 Write unit tests for find_text/find_icon against fixture Scene objects (found and not-found cases)

4.1 Implement runtime/context.py: per-task in-memory context holding Scene history and executed step results
4.2 Implement runtime/planner.py: given a goal + current Scene (+ context), produce an ordered list of intended next steps
4.3 Implement runtime/executor.py: turn a planned step into tools/ calls, with retry/backoff and wait-for-element handling, bounded by configurable max-retries and max-steps
4.4 Implement runtime/task.py: drive the Observe→Think→Act→Observe loop for a task from start to completion/failure, wiring Planner + Executor + Context together
4.5 Add max-step/max-retry ceiling handling that fails the task with a clear reason instead of looping indefinitely
4.6 Write unit tests for Executor retry/backoff behavior using a fake tool that fails N times then succeeds
4.7 Write an end-to-end test of the loop against a mocked Driver/Scene sequence simulating "open app → find search → tap → type"

5.1 Implement storage/artifact_store.py: write screenshots and per-step JSON (Scene, prompt, tool call, result) to tasks/history/<task_id>/NNN.{png,json}
5.2 Implement storage/timeline.py: append-and-read API for a task's ordered step records, backed by artifact_store
5.3 Add SQLite-backed task metadata storage (task id, device id, status, start/end timestamps) with create/update/query functions
5.4 Wire runtime/task.py (from section 4) to write a timeline record after every Executor step and update task metadata on completion/failure
5.5 Write unit tests verifying timeline records are written in order and are readable after a simulated process restart (re-opening the same task directory)

6.1 Implement api/mcp.py: register take_screenshot, tap, swipe, input_text, launch_app, find_text, find_icon, get_ui_tree, describe_screen, list_devices, device_status as MCP tools calling directly into tools/
6.2 Add a semantic error-translation layer so driver/framework errors (WDA connection errors, element-not-found) surface as clear MCP tool errors (e.g. "device offline", "element not found")
6.3 Implement api/rest.py (FastAPI) with GET /devices, POST /devices/{id}/tap, POST /devices/{id}/screenshot, POST /devices/{id}/launch, POST /agent/task, GET /task/{id}, calling the same tools//runtime/ functions as the MCP tools
6.4 Write a test that starts a task via POST /agent/task and polls GET /task/{id} until it completes (using a mocked device)
6.5 Write a test that calls each MCP tool against a mocked device and asserts no Appium/WDA-specific type or string leaks into the response

7.1 Run the full Observe→Think→Act→Observe loop against a real (or simulator) iPhone for a simple goal (e.g. "open an app and search for a term"), verifying tap/input/screenshot all work through the real WDADriver
7.2 Confirm the resulting task's timeline directory contains a complete, ordered set of per-step screenshots and JSON records
7.3 Confirm an MCP client (or a manual MCP tool call harness) can drive the same real-device flow end-to-end through api/mcp.py