# Device Agent Runtime

Device Agent Runtime is a device-agnostic runtime for LLM-driven automation.
It gives agents a stable way to observe, decide, and act against real devices
through a small set of domain models, driver contracts, tools, perception
providers, and runtime orchestration.

iPhone automation through WebDriverAgent/Appium is the first driver, not the
platform boundary. Future drivers can target Android, browsers, desktop
environments, or other device surfaces without changing the runtime's core
contracts.

## Current Shape

- `core/`: shared domain models and runtime errors.
- `driver/`: the `Driver` contract, concrete driver adapters, and driver-type
  registry.
- `device/`: device lifecycle and active driver management.
- `tools/`: device capabilities exposed to Runtime and adapter layers.
- `perception/`: screen-to-`Scene` perception behind `PerceptionProvider`.
- `runtime/`: planning and execution orchestration.
- `api/`: MCP and supporting integration adapters.
- `storage/`: timeline, task, and device configuration persistence.
- `packages/cloud-platform/`: cloud scheduling, device pooling, plugins, and
  the Python cloud SDK as the `device-cloud-platform` workspace member.
- `apps/cloud-api/`: deployable authenticated Cloud Control Plane with
  PostgreSQL/SQLite persistence, scheduling, leases, and health endpoints.
- `apps/device-host-agent/`: outbound Host Agent that synchronizes configured
  devices and executes leased tasks through the existing Runtime/workflow.

## Python Workspace

The repository uses a uv workspace with one committed lockfile. From the
repository root, synchronize every Python member with:

```bash
uv sync --locked --all-packages
```

Run the complete local test suite in a workspace environment:

```bash
uv run --all-packages pytest -m "not integration"
```

Select one member when running package-specific commands:

```bash
uv run --package device-agent-runtime python -c "import runtime"
uv run --package device-cloud-platform python -c "import cloud"
uv run --package device-cloud-api device-cloud-api --help
uv run --package device-host-agent device-host-agent --help
uv build --package device-agent-runtime
uv build --package device-cloud-platform
```

Runtime is an in-process execution library, not a standalone HTTP service. The
Host Agent console at `http://127.0.0.1:8765/tasks` is the authenticated
operator view for the tasks that actually execute on that Host, including
per-step screenshots, OCR observations, and UI-tree results. The Cloud Console
remains the fleet-level view for dispatch status and Cloud-proxy planner history.

## Local-only Host Agent

Run without a Cloud Control Plane by setting `HOST_AGENT_MODE=local`. Tasks
submitted in the Host console are queued and executed in the same process:

```bash
export HOST_AGENT_MODE=local
export AI_PLANNER_ENABLED=true
export AI_PLANNER_PROVIDER=openai-compatible
export AI_PLANNER_MODEL=qwen2.5
export AI_PLANNER_API_KEY=local-key
export AI_PLANNER_BASE_URL=http://127.0.0.1:11434/v1
uv run --package device-host-agent device-host-agent setup
uv run --package device-host-agent device-host-agent
```

`openai-compatible` works with Ollama, LM Studio, vLLM, or another server that
implements OpenAI `/chat/completions`. Hosted `openai` and `anthropic` providers
also accept `AI_PLANNER_API_KEY` and their conventional API key variables.

The local Host console also exposes an authenticated conversational Agent API:

```text
POST http://127.0.0.1:8765/api/chat
```

Send a JSON body containing `messages` (`user`/`assistant` roles). The Agent can
return ordinary assistant text or call the same device-operation tool contracts
used by the Runtime; each tool result is fed back to the model before the final
reply is returned. The endpoint uses the Host console session cookie, so it is
not an unauthenticated device-control endpoint.

For vision-capable OpenAI models, a user message may contain standard OpenAI
multimodal blocks:

```json
{
  "messages": [{
    "role": "user",
    "content": [
      {"type": "text", "text": "点击图片中的登录按钮"},
      {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
    ]
  }]
}
```

The compact form `{ "role": "user", "text": "...", "image_base64": "..." }`
is also accepted. The model can inspect the supplied image and then call a
phone tool such as `tap` in the same conversation.

Local mode records LLM responses, reasoning fields, tool calls, tool results,
and final replies as JSONL in `host_agent_data/conversations.jsonl`. Image
bytes are excluded; set `HOST_AGENT_CONVERSATION_LOG_PATH` to change the path.

In local mode, Appium supervision is enabled by default. Host Agent probes
`/status`, adopts a healthy existing Appium instance, starts Appium when no
listener exists, restarts only processes it started if they crash, and stops
those child processes on shutdown. Override `HOST_AGENT_APPIUM_*` or set
`HOST_AGENT_DEPENDENCY_SUPERVISOR_ENABLED=false` when an external process
manager owns Appium.

## Project Direction

The durable roadmap is in [docs/ROADMAP.md](docs/ROADMAP.md). The architecture
invariants future changes must preserve are in
[docs/CONSTITUTION.md](docs/CONSTITUTION.md).

## Operator Guides

- [Cloud Control Plane deployment](docs/CLOUD_DEPLOYMENT.md): run local SQLite
  or deployed PostgreSQL, configure credentials and Runtime AI planning, and
  perform orderly shutdown or rollback.
- [macOS migration and real iPhone setup](docs/MACOS_IPHONE_SETUP.md): install
  Xcode, Appium/XCUITest, sign WebDriverAgent, verify a real device, and run a
  connected Host Agent.