agentic-mobile-control/openspec/changes/apex-agent-mvp/tasks.md

## 1. Project scaffolding

- [x] 1.1 Create the package layout: `core/`, `tools/`, `vision/`, `runtime/`, `api/`, `storage/`, `tests/` (each with `__init__.py`)
- [x] 1.2 Add MVP dependencies to `pyproject.toml` (Appium Python client, an OCR engine e.g. PaddleOCR, FastAPI, an MCP server SDK, SQLite driver, test framework)
- [x] 1.3 Add `core/models.py` with shared data classes: `Device`, `Scene`, `SceneElement`, `Task`, `Step`
- [x] 1.4 Set up a basic test runner config and a smoke test that imports every new package

## 2. Device management (capability: device-management)

- [x] 2.1 Define the abstract `Driver` interface in `core/driver.py` (`connect`, `disconnect`, `screenshot`, `tap`, `swipe`, `input`, `launch`, `terminate`, `tree`, `home`, `lock`, `unlock`) with no stored business state
- [x] 2.2 Implement `core/device_manager.py`: device discovery/registration, status tracking (`idle`/`busy`/`offline`/`error`), `list_devices()`, `connect()`, `disconnect()`, `status()`
- [x] 2.3 Implement `core/wda_driver.py` (`WDADriver`) using the Appium Python client against WebDriverAgent, covering all `Driver` methods
- [x] 2.4 Add connection-loss handling: mark device `offline`/`error` and bound retries on `connect()` instead of hanging
- [x] 2.5 Write unit tests for `DeviceManager` state transitions using a fake/mock `Driver`
- [x] 2.6 Write an integration test (skippable without hardware) that connects to a real/simulated device via `WDADriver` and takes a screenshot

## 3. Perception pipeline (capability: scene-perception)

- [x] 3.1 Implement `vision/ocr.py`: run OCR over a screenshot and return text boxes with bounds/confidence
- [x] 3.2 Implement `vision/ui_parser.py`: parse the raw UI tree returned by `Driver.tree()` into a normalized element list (type, text, bounds)
- [x] 3.3 Implement `vision/scene_builder.py`: fuse OCR output + parsed UI tree into a single `Scene`, deduping overlapping elements by bounding-box IoU (prefer UI-tree bounds/type on overlap)
- [x] 3.4 Implement `vision/icon_detector.py` as a minimal stub (template match or "not found") satisfying the `find_icon` contract
- [x] 3.5 Implement `tools/screenshot.py`, `tools/tap.py`, `tools/swipe.py`, `tools/input_text.py`, `tools/launch_app.py`, `tools/ui_tree.py` as thin wrappers over the active device's `Driver`
- [x] 3.6 Implement `tools/describe_screen.py` (returns the fused `Scene`) and `find_text`/`find_icon` helpers that search a `Scene` and return coordinates or "not found"
- [x] 3.7 Write unit tests for `scene_builder` dedup logic using fixture screenshots/tree/OCR data
- [x] 3.8 Write unit tests for `find_text`/`find_icon` against fixture `Scene` objects (found and not-found cases)

## 4. Agent runtime (capability: agent-runtime)

- [x] 4.1 Implement `runtime/context.py`: per-task in-memory context holding Scene history and executed step results
- [x] 4.2 Implement `runtime/planner.py`: given a goal + current `Scene` (+ context), produce an ordered list of intended next steps
- [x] 4.3 Implement `runtime/executor.py`: turn a planned step into `tools/` calls, with retry/backoff and wait-for-element handling, bounded by configurable max-retries and max-steps
- [x] 4.4 Implement `runtime/task.py`: drive the Observe→Think→Act→Observe loop for a task from start to completion/failure, wiring Planner + Executor + Context together
- [x] 4.5 Add max-step/max-retry ceiling handling that fails the task with a clear reason instead of looping indefinitely
- [x] 4.6 Write unit tests for `Executor` retry/backoff behavior using a fake tool that fails N times then succeeds
- [x] 4.7 Write an end-to-end test of the loop against a mocked `Driver`/`Scene` sequence simulating "open app → find search → tap → type"

## 5. Task memory / timeline (capability: task-memory)

- [x] 5.1 Implement `storage/artifact_store.py`: write screenshots and per-step JSON (Scene, prompt, tool call, result) to `tasks/history/<task_id>/NNN.{png,json}`
- [x] 5.2 Implement `storage/timeline.py`: append-and-read API for a task's ordered step records, backed by `artifact_store`
- [x] 5.3 Add SQLite-backed task metadata storage (task id, device id, status, start/end timestamps) with create/update/query functions
- [x] 5.4 Wire `runtime/task.py` (from section 4) to write a timeline record after every Executor step and update task metadata on completion/failure
- [x] 5.5 Write unit tests verifying timeline records are written in order and are readable after a simulated process restart (re-opening the same task directory)

## 6. MCP tool server & REST API (capability: mcp-tool-server)

- [x] 6.1 Implement `api/mcp.py`: register `take_screenshot`, `tap`, `swipe`, `input_text`, `launch_app`, `find_text`, `find_icon`, `get_ui_tree`, `describe_screen`, `list_devices`, `device_status` as MCP tools calling directly into `tools/`
- [x] 6.2 Add a semantic error-translation layer so driver/framework errors (WDA connection errors, element-not-found) surface as clear MCP tool errors (e.g. "device offline", "element not found")
- [x] 6.3 Implement `api/rest.py` (FastAPI) with `GET /devices`, `POST /devices/{id}/tap`, `POST /devices/{id}/screenshot`, `POST /devices/{id}/launch`, `POST /agent/task`, `GET /task/{id}`, calling the same `tools/`/`runtime/` functions as the MCP tools
- [x] 6.4 Write a test that starts a task via `POST /agent/task` and polls `GET /task/{id}` until it completes (using a mocked device)
- [x] 6.5 Write a test that calls each MCP tool against a mocked device and asserts no Appium/WDA-specific type or string leaks into the response

## 7. End-to-end validation

- [ ] 7.1 Run the full Observe→Think→Act→Observe loop against a real (or simulator) iPhone for a simple goal (e.g. "open an app and search for a term"), verifying tap/input/screenshot all work through the real `WDADriver`
- [ ] 7.2 Confirm the resulting task's timeline directory contains a complete, ordered set of per-step screenshots and JSON records
- [ ] 7.3 Confirm an MCP client (or a manual MCP tool call harness) can drive the same real-device flow end-to-end through `api/mcp.py`