Device Agent Runtime is a device-agnostic runtime for LLM-driven automation. It gives agents a stable way to observe, decide, and act against real devices through a small set of domain models, driver contracts, tools, perception providers, and runtime orchestration.

iPhone automation through WebDriverAgent/Appium is the first driver, not the platform boundary. Future drivers can target Android, browsers, desktop environments, or other device surfaces without changing the runtime's core contracts.

Current Shape

core/: shared domain models and runtime errors.
driver/: the Driver contract, concrete driver adapters, and driver-type registry.
device/: device lifecycle and active driver management.
tools/: device capabilities exposed to Runtime and adapter layers.
perception/: screen-to-Scene perception behind PerceptionProvider.
runtime/: planning and execution orchestration.
api/: MCP and supporting integration adapters.
storage/: timeline, task, and device configuration persistence.
packages/cloud-platform/: cloud scheduling, device pooling, plugins, and the Python cloud SDK as the device-cloud-platform workspace member.
apps/cloud-api/: deployable authenticated Cloud Control Plane with PostgreSQL/SQLite persistence, scheduling, leases, and health endpoints.
apps/device-host-agent/: outbound Host Agent that synchronizes configured devices and executes leased tasks through the existing Runtime/workflow.

Python Workspace

The repository uses a uv workspace with one committed lockfile. From the repository root, synchronize every Python member with:

uv sync --locked --all-packages

Run the complete local test suite in a workspace environment:

uv run --all-packages pytest -m "not integration"

Select one member when running package-specific commands:

uv run --package device-agent-runtime python -c "import runtime"
uv run --package device-cloud-platform python -c "import cloud"
uv run --package device-cloud-api device-cloud-api --help
uv run --package device-host-agent device-host-agent --help
uv build --package device-agent-runtime
uv build --package device-cloud-platform

Runtime is an in-process execution library, not a standalone HTTP service. The Host Agent console at http://127.0.0.1:8765/tasks is the authenticated operator view for the tasks that actually execute on that Host, including per-step screenshots, OCR observations, and UI-tree results. The Cloud Console remains the fleet-level view for dispatch status and Cloud-proxy planner history.

Local-only Host Agent

Run without a Cloud Control Plane by setting HOST_AGENT_MODE=local. Tasks submitted in the Host console are queued and executed in the same process:

export HOST_AGENT_MODE=local
export AI_PLANNER_ENABLED=true
export AI_PLANNER_PROVIDER=openai-compatible
export AI_PLANNER_MODEL=qwen2.5
export AI_PLANNER_API_KEY=local-key
export AI_PLANNER_BASE_URL=http://127.0.0.1:11434/v1
export AI_PLANNER_MULTIMODAL=true
uv run --package device-host-agent device-host-agent setup
uv run --package device-host-agent device-host-agent

openai-compatible works with Ollama, LM Studio, vLLM, or another server that implements OpenAI /chat/completions. Hosted openai and anthropic providers also accept AI_PLANNER_API_KEY and their conventional API key variables.

The local Host console also exposes an authenticated conversational Agent API:

POST http://127.0.0.1:8765/api/chat

Send a JSON body containing device_id and messages (user/assistant roles). The Agent can return ordinary assistant text or call the same device-operation tool contracts used by the Runtime; each tool result is fed back to the model before the final reply is returned. The endpoint uses the Host console session cookie, so it is not an unauthenticated device-control endpoint.

Every chat request is bound to exactly one registered device_id. Keep a separate message history and client session for each phone; the Agent injects the bound device into device tools and rejects cross-device tool arguments.

For vision-capable OpenAI models, a user message may contain standard OpenAI multimodal blocks:

{
  "messages": [{
    "role": "user",
    "content": [
      {"type": "text", "text": "点击图片中的登录按钮"},
      {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
    ]
  }]
}

The compact form { "role": "user", "text": "...", "image_base64": "..." } is also accepted. The model can inspect the supplied image and then call a phone tool such as tap in the same conversation.

Set AI_PLANNER_MULTIMODAL=true for a vision model. The Planner sends the screenshot and omits OCR-only elements and OCR metadata from the structured scene payload, avoiding duplicate OCR text.

Local mode records LLM responses, reasoning fields, tool calls, tool results, and final replies in a local SQLite database. View them at http://127.0.0.1:8765/conversations; image bytes are excluded. Set HOST_AGENT_CONVERSATION_LOG_PATH to change the database path.

In local mode, Appium supervision is enabled by default. Host Agent probes /status, adopts a healthy existing Appium instance, starts Appium when no listener exists, restarts only processes it started if they crash, and stops those child processes on shutdown. Override HOST_AGENT_APPIUM_* or set HOST_AGENT_DEPENDENCY_SUPERVISOR_ENABLED=false when an external process manager owns Appium.

Project Direction

The durable roadmap is in docs/ROADMAP.md. The architecture invariants future changes must preserve are in docs/CONSTITUTION.md.

Operator Guides

Cloud Control Plane deployment: run local SQLite or deployed PostgreSQL, configure credentials and Runtime AI planning, and perform orderly shutdown or rollback.
macOS migration and real iPhone setup: install Xcode, Appium/XCUITest, sign WebDriverAgent, verify a real device, and run a connected Host Agent.