q792602257
7f439f0db5
Host-agent console showed OCR/UI-tree overlay boxes misaligned with the displayed screenshot. Two independent causes, both confirmed with real task data and pixel-level measurement of a user-provided screenshot: 1. perception/ui_parser.py parses XCUITest UI-tree bounds as iOS logical points, while scene_builder.py's Scene.width/height (via infer_png_size) and OCR bounds are in screenshot pixels, never reconciled (2.0x on Retina devices). build_scene() now detects the scale from the first x==0,y==0 UI element and rescales OCR bounds down to points-space, reporting Scene.width/height in points too. No-op for Android, where UiAutomator2 bounds already match pixels 1:1. This also fixes tap() landing at the wrong location for OCR-matched text, and lets the IOU fusion between UI-tree and OCR elements actually fire on iOS. 2. runtime/task.py captured `scene` (OCR/UI-tree data) before the LLM planning call, but re-captured `before_screenshot` for each step afterward - a real time gap during which on-screen content (e.g. a keyboard) could shift, producing a directional drift between the overlay and the displayed image. The first step of each plan batch now reuses the screenshot already taken for planning instead of capturing a new one; later steps in a multi-step batch still take a fresh capture (left unresolved, scoped out by request). Regression tests added for both the scale reconciliation (using real 828x1792 vs 414x896 numbers) and the screenshot reuse behavior.
Device Agent Runtime
Device Agent Runtime is a device-agnostic runtime for LLM-driven automation. It gives agents a stable way to observe, decide, and act against real devices through a small set of domain models, driver contracts, tools, perception providers, and runtime orchestration.
iPhone automation through WebDriverAgent/Appium is the first driver, not the platform boundary. Future drivers can target Android, browsers, desktop environments, or other device surfaces without changing the runtime's core contracts.
Current Shape
core/: shared domain models and runtime errors.driver/: theDrivercontract, concrete driver adapters, and driver-type registry.device/: device lifecycle and active driver management.tools/: device capabilities exposed to Runtime and adapter layers.perception/: screen-to-Sceneperception behindPerceptionProvider.runtime/: planning and execution orchestration.api/: MCP and supporting integration adapters.storage/: timeline, task, and device configuration persistence.packages/cloud-platform/: cloud scheduling, device pooling, plugins, and the Python cloud SDK as thedevice-cloud-platformworkspace member.apps/cloud-api/: deployable authenticated Cloud Control Plane with PostgreSQL/SQLite persistence, scheduling, leases, and health endpoints.apps/device-host-agent/: outbound Host Agent that synchronizes configured devices and executes leased tasks through the existing Runtime/workflow.
Python Workspace
The repository uses a uv workspace with one committed lockfile. From the repository root, synchronize every Python member with:
uv sync --locked --all-packages
Run the complete local test suite in a workspace environment:
uv run --all-packages pytest -m "not integration"
Select one member when running package-specific commands:
uv run --package device-agent-runtime python -c "import runtime"
uv run --package device-cloud-platform python -c "import cloud"
uv run --package device-cloud-api device-cloud-api --help
uv run --package device-host-agent device-host-agent --help
uv build --package device-agent-runtime
uv build --package device-cloud-platform
Runtime is an in-process execution library, not a standalone HTTP service. The
Host Agent console at http://127.0.0.1:8765/tasks is the authenticated
operator view for the tasks that actually execute on that Host, including
per-step screenshots, OCR observations, and UI-tree results. The Cloud Console
remains the fleet-level view for dispatch status and Cloud-proxy planner history.
Project Direction
The durable roadmap is in docs/ROADMAP.md. The architecture invariants future changes must preserve are in docs/CONSTITUTION.md.
Operator Guides
- Cloud Control Plane deployment: run local SQLite or deployed PostgreSQL, configure credentials and Runtime AI planning, and perform orderly shutdown or rollback.
- macOS migration and real iPhone setup: install Xcode, Appium/XCUITest, sign WebDriverAgent, verify a real device, and run a connected Host Agent.