q792602257andClaude Opus 4.6
75879c8a52
Tests / Test failed: 2, passed: 691
Adds an opt-in dependency supervisor inside the Host Agent that probes,
spawns, and restarts the two local processes the macOS single-machine
real-device workflow depends on: the Appium server (gates Driver.connect())
and the local Runtime API (local inspection). Default-off; gated by
HOST_AGENT_DEPENDENCY_SUPERVISOR_ENABLED plus per-dependency *_SUPERVISED
flags.
Mitigates the live-incident failure mode where forgetting to start Appium
silently keeps devices offline and tasks queued forever with no error
surfaced in Host Agent logs.
Behavior (per openspec change):
- Adopt-don't-fight: probe (TCP + dependency-specific HTTP health check)
before spawn. Healthy listener → adopted (never killed/restarted).
Unhealthy listener → port-conflict error, skip. No listener → spawn.
- Only supervisor-spawned processes are restarted on crash, with capped
exponential backoff (1s/2s/4s/8s, capped at 30s) and a per-process-lifetime
attempt ceiling (HOST_AGENT_DEPENDENCY_RESTART_MAX_ATTEMPTS, default 5).
- Spawn failures (e.g. missing executable) logged distinctly from crashes.
- Graceful stop terminates only spawned children; adopted processes untouched.
- Supervisor starts before the heartbeat loop's first connect_devices() pass
and stops alongside existing heartbeat/console teardown.
Validation: ruff check + format clean, compileall clean, openspec validate
--strict valid. Non-integration suite 503 passed / 44 deselected / 2 failed
(both failures pre-existing from unrelated 03c7c30 LLM_PROVIDER_ENC_KEY;
verified by stashing this change). macOS real-device manual verification
(task 6.4) deferred to a macOS host.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
75 lines
12 KiB
Markdown
75 lines
12 KiB
Markdown
## Context
|
|
|
|
The Host Agent (`apps/device-host-agent/host_agent/`) is a headless outbound worker: `HostAgentApplication.run_async()` (`app.py:41-90`) runs a heartbeat loop and a claim/execute loop, and (since `host-agent-local-console`) an embedded local web console bound to `127.0.0.1:8765` by default. None of this touches Appium or the local Runtime API.
|
|
|
|
Real device control depends on two processes the Host Agent does not manage today:
|
|
- **Appium server**, default `http://127.0.0.1:4723` (`docs/MACOS_IPHONE_SETUP.md:209`, `driver/wda_driver.py:12`, `driver/android_driver.py:22`). `Driver.connect()` opens a `webdriver.Remote` session against it; on failure it raises `DeviceOfflineError`, which `DeviceManager.connect()` (`device/manager.py:68-108`) catches and turns into device status `"offline"`/`"error"` — this is exactly what happened in the live incident that motivated this change: Appium wasn't running, the device stayed offline, and every assigned task queued forever with no error surfaced anywhere.
|
|
- **Local Runtime API** (`api.rest:create_app`, default `127.0.0.1:8000`), a fully separate FastAPI app/process used to inspect device/task/world state locally. It shares no runtime coupling with the Host Agent process today ("Runtime 不依赖/打包 cloud 或 application" — established repo convention) and this change does not introduce any.
|
|
|
|
Appium already owns the entire WDA session lifecycle end to end (`docs/MACOS_IPHONE_SETUP.md:176`, `:455` — "本项目会让 Appium 的 XCUITest Driver 创建和管理 WDA session"; "常规单机使用优先让 Appium 管理 WDA，不要一开始就引入 iproxy 或手工 WDA 生命周期"). Any supervision the Host Agent adds must sit *outside* that boundary — it manages whether the Appium **process** is running, never what Appium does internally with WDA.
|
|
|
|
## Goals / Non-Goals
|
|
|
|
**Goals:**
|
|
- Let the Host Agent optionally start Appium and the local Runtime API as plain subprocesses when they aren't already running, so a forgotten manual step turns into "it just works" instead of "device silently offline forever."
|
|
- Make the current state (spawned vs. adopted-existing vs. failed-to-start) visible in the Host Agent's own log/console, so this remains debuggable rather than adding a second silent-failure mode on top of the one it fixes.
|
|
- Never conflict with an already-running instance: detect and adopt (read-only) rather than double-spawn or steal a port.
|
|
|
|
**Non-Goals:**
|
|
- No change to Appium's ownership of WDA session lifecycle — the supervisor never touches WDA directly, never passes `webDriverAgentUrl`, never signs/builds anything.
|
|
- No Docker/Compose story. `compose.yaml`/`compose.deploy.yaml` are untouched; this is native-process-only, scoped to the macOS single-machine real-device workflow described in `docs/MACOS_IPHONE_SETUP.md`.
|
|
- No change to `driver/registry.py`, `driver/wda_driver.py`, `driver/android_driver.py`, or the outbound cloud protocol (`host-agent-protocol`).
|
|
- No process supervision beyond these two known dependencies (not a general-purpose process manager).
|
|
- No remote/cloud configuration of this feature — it's a local opt-in, matching how `host_agent/config.py` is already entirely env-var driven per-Host.
|
|
|
|
## Decisions
|
|
|
|
**1. Default: entire supervisor is off (`HOST_AGENT_DEPENDENCY_SUPERVISOR_ENABLED=false`).**
|
|
Unlike `host-agent-local-console` (which is now forced-on because it's a passive, loopback-bound, low-risk read/manage surface), this feature actively spawns and can restart external processes — including, in the crash-restart path, potentially interrupting an in-flight Appium session/WDA connection if a spawned Appium process is killed and restarted mid-task. That's a materially different risk profile from serving a status page, so it defaults off and is opt-in per Host, consistent with how `AI_PLANNER_ENABLED` defaulting decisions in this repo have been made deliberately per-surface rather than blanket. Each of the two dependencies (Appium, Runtime) also has its own enable flag nested under the top-level flag, so an operator can supervise Appium only, Runtime only, or both.
|
|
|
|
**2. Adopt-don't-fight: TCP connect + HTTP health probe before spawn.**
|
|
Before spawning either process, the supervisor does a plain TCP connect to `(host, port)`. If something answers, it issues the same health check used post-spawn (Appium: `GET /status` expects HTTP 200 with `ready`/build JSON per `docs/MACOS_IPHONE_SETUP.md:218`; Runtime: `GET /health` or equivalent existing endpoint on `api.rest`). If the health check passes, the supervisor logs "adopted existing instance at host:port" and does **not** spawn a subprocess, does **not** track it as "ours," and will never kill or restart it. If the port is occupied but the health check fails (something else is listening, or it's an unhealthy/half-started instance), the supervisor logs an error for that dependency and leaves it un-started rather than guessing — port conflicts get surfaced, not silently resolved. This directly implements the concern raised during proposal review that an already-running manually-started Appium must not be duplicated or fought over.
|
|
|
|
**3. Only supervisor-spawned processes are restarted; adopted processes are never touched.**
|
|
This is the direct mitigation for the WDA-session-interruption risk: an operator who already has their own long-lived Appium running (e.g., manually, or from a previous supervised run in another terminal) is never at risk of the Host Agent killing/restarting it. Restart-on-crash only applies to a `subprocess.Popen` handle the supervisor itself holds.
|
|
|
|
**4. Crash-restart uses capped exponential backoff with a per-run attempt ceiling.**
|
|
Backoff schedule (e.g. 1s, 2s, 4s, 8s, capped at 30s) with a max attempt count per dependency per Host Agent process lifetime (e.g. 5); once exhausted, the supervisor logs that it's giving up on that dependency and stops trying, rather than restart-looping forever. This directly answers the "限流/退避避免死循环重启风暴" concern. The counter resets only on a fresh Host Agent process start, not periodically — a persistently-crashing Appium install shouldn't be retried indefinitely in the background.
|
|
|
|
**5. Spawn mechanism: stdlib `subprocess.Popen`, no new dependency.**
|
|
Appium is invoked as `appium --address <host> --port <port>` (assumes `appium` is already on `PATH`, exactly as the manual instructions already require — this change does not install or vendor Appium). Runtime is invoked as the existing `uvicorn api.rest:create_app --factory --host <host> --port <port>` command, run as a subprocess rather than in-process, preserving the existing "Runtime is a fully separate process" boundary — this is process supervision, not a code-level dependency between the two packages. Both children's stdout/stderr are captured and forwarded into the Host Agent's own logging (prefixed per-process) so a crash's root cause doesn't require the operator to have had a spare terminal open.
|
|
|
|
**6. Config fields follow the existing `HostAgentConfig`/`HOST_AGENT_*` convention.**
|
|
New fields on `HostAgentConfig` (mirroring the `console_*` fields added by `host-agent-local-console`):
|
|
- `dependency_supervisor_enabled: bool` ← `HOST_AGENT_DEPENDENCY_SUPERVISOR_ENABLED` (default `false`)
|
|
- `appium_supervised: bool` ← `HOST_AGENT_APPIUM_SUPERVISED` (default `false`)
|
|
- `appium_host: str` ← `HOST_AGENT_APPIUM_HOST` (default `127.0.0.1`)
|
|
- `appium_port: int` ← `HOST_AGENT_APPIUM_PORT` (default `4723`, matching the documented default)
|
|
- `runtime_supervised: bool` ← `HOST_AGENT_RUNTIME_SUPERVISED` (default `false`)
|
|
- `runtime_host: str` ← `HOST_AGENT_RUNTIME_HOST` (default `127.0.0.1`)
|
|
- `runtime_port: int` ← `HOST_AGENT_RUNTIME_PORT` (default `8000`, matching the documented default)
|
|
- `dependency_restart_max_attempts: int` ← `HOST_AGENT_DEPENDENCY_RESTART_MAX_ATTEMPTS` (default `5`)
|
|
|
|
**7. Wiring point: alongside the existing heartbeat/claim loop, not inside it.**
|
|
`HostAgentApplication` starts the supervisor (if enabled) before starting the heartbeat loop and `connect_devices()` pass, and stops it during shutdown alongside the existing console/heartbeat teardown in `run_async()` — so a supervised Appium is up before the first `DeviceManager.connect()` attempt in the same startup sequence that already exists.
|
|
|
|
## Risks / Trade-offs
|
|
|
|
- **[Risk] A supervisor-spawned Appium restart mid-task interrupts an active WDA session, failing whatever task is running.** → Mitigation: restart only triggers on the spawned process actually exiting (crash), not on a health-check hiccup; the underlying task still fails cleanly through the existing `DeviceOfflineError` path (same as today when Appium isn't running at all) rather than in some new/undefined way. This is a pre-existing failure mode (Appium can already crash unsupervised) — the difference is the Host Agent now also tries to bring it back instead of leaving the device offline until a human notices.
|
|
- **[Risk] Supervising the Runtime API blurs the "fully independent process" boundary this repo has deliberately maintained.** → Mitigation: it's spawned as a subprocess via its existing CLI entry point, not imported/mounted in-process — no new Python-level dependency from `device-host-agent` onto `api.rest`, no shared code path. The boundary that matters (no import coupling, no shared FastAPI app) is preserved; only process *lifecycle* (start/stop timing) is now optionally shared.
|
|
- **[Risk] `appium` not on `PATH`, or wrong Appium driver installed (per `docs/MACOS_IPHONE_SETUP.md` xcuitest/uiautomator2 driver setup).** → Mitigation: spawn failure (e.g. `FileNotFoundError` from `subprocess.Popen`) is logged clearly as a dependency-supervisor error, distinct from a normal Appium crash, so the operator isn't left thinking Appium crashed when it never started.
|
|
- **[Risk] Health-check false positive: something unrelated happens to be listening on 4723/8000.** → Mitigation: the health check requires the actual expected response shape (Appium `/status` JSON, Runtime health endpoint), not just "port is open" — an unrelated listener fails the check and is treated as a conflict, not adopted.
|
|
- **[Trade-off] This only covers macOS native real-device usage; Docker/CI paths get no benefit.** → Accepted per proposal scope — those paths don't have this failure mode in the first place (no interactive multi-terminal manual step to forget).
|
|
|
|
## Migration Plan
|
|
|
|
- Fully additive and opt-in: existing Host Agent deployments/configs are unaffected with zero config changes (all new fields default to off/disabled).
|
|
- Rollout is per-Host: an operator turns it on by setting `HOST_AGENT_DEPENDENCY_SUPERVISOR_ENABLED=true` (and the relevant `*_SUPERVISED` flags) in their existing `.env`/environment, no schema or protocol changes involved.
|
|
- Rollback: unset the env vars (or set back to `false`) and restart the Host Agent — reverts to today's fully-manual behavior with no residual state (the supervisor holds no persistent storage).
|
|
- Documentation: `docs/MACOS_IPHONE_SETUP.md` gains a section presenting supervised mode as an optional alternative alongside the existing manual steps, which remain the documented default path.
|
|
|
|
## Open Questions
|
|
|
|
- Should `dependency_restart_max_attempts` exhaustion eventually surface in the local console (`host-agent-local-console`) status page rather than only in logs? Left for a future change if operators find log-only insufficient in practice.
|
|
- Android/UiAutomator2 real-device validation for this supervisor is out of scope for the same reason `android-driver` itself shipped without real-device validation — should be revisited once Android hardware validation happens.
|