q792602257andClaude Sonnet 5
bb6f24bbcb
This change was originally proposed as the root-cause fix for the
2026-07-14 DeviceNotFoundError incident. That diagnosis was wrong: it was
subsequently confirmed only one Host Agent process was running at the time,
ruling out the duplicate-process precondition this change addresses. The
actual root cause was execution.py's create_task_runner() omitting manager=
when wiring TaskRunner (see 08cef7c). Reframe the Why section: this change
stands on its own as independent duplicate-process hardening, not as a fix
for an incident it turned out not to have caused.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
28 lines
4.8 KiB
Markdown
28 lines
4.8 KiB
Markdown
## Why
|
|
|
|
Duplicate Host Agent processes sharing the same host identity are a real, unguarded risk: a repo-wide search confirms no pidfile, lockfile, or `flock`-style guard exists anywhere in `apps/device-host-agent`, and both `identity_path` and the device-config store default to process-working-directory-relative paths, which makes an accidental duplicate launch easy to trigger unnoticed (e.g. a supervisor starting a replacement before a hung previous instance has fully exited, or an operator accidentally leaving a second instance running in another terminal/tmux pane). If it ever happens, the failure mode is severe and confusing: `DeviceManager` (`device/manager.py`) is in-process, in-memory, and per-process, with no persistence and no reactive removal, while `TaskScheduler.assign()` (`cloud/scheduler.py`) dispatches to a `host_id` based on a periodically-synced `DevicePool` snapshot with no concept of *which OS process* is currently answering that host's long-poll — both instances would heartbeat and claim under the same `host_id`, and a claim could land on whichever instance answers next, including one whose own in-memory `DeviceManager` never registered the device.
|
|
|
|
This change was originally proposed as the root-cause fix for a production incident (`DeviceNotFoundError: unknown device device-8967d09f0d5f4cf49f7361a9f0dcb0ce`, host `host-311f68...`, 2026-07-14). That diagnosis was **wrong**: it was subsequently confirmed that only one Host Agent process was running at the time, which rules out the duplicate-process precondition this change addresses. The actual root cause was `create_task_runner()` (`apps/device-host-agent/host_agent/execution.py`) omitting `manager=` when wiring `TaskRunner`'s `observer`/`screenshot_provider`, so both silently fell back to the process-global `DEFAULT_MANAGER` singleton instead of the Host Agent's real, device-populated `DeviceManager` — fixed directly in code, without an OpenSpec proposal, since it was a small, fully-diagnosed, single-file wiring bug. This proposal is kept anyway, on its own merits: duplicate-process protection is a legitimate, independent hardening measure against a real gap the investigation surfaced along the way, not a fix for an incident it turned out not to have caused.
|
|
|
|
## What Changes
|
|
|
|
- `create_application()` (`apps/device-host-agent/host_agent/app.py:145`) acquires a local, exclusive, non-blocking instance lock as its first action, before `resolve_host_identity()` or any device-enrollment network call.
|
|
- The lock file is colocated with the existing per-installation state directory (`identity_path.parent` — the same directory `app.py` already uses for `host_console_history.sqlite3` and `host_governance_policy.json`), so this requires no new configuration surface.
|
|
- If the lock is already held by a live process, the Host Agent logs a clear, actionable "duplicate instance" error (distinct from a generic startup failure) and exits non-zero without contacting the control plane, enrolling devices, or starting the console/heartbeat/claim loops.
|
|
- The lock releases automatically on process exit for any reason — clean shutdown, unhandled exception, or a forced kill — via OS-level advisory file locking rather than a pidfile/sentinel-existence check, so no stale-lock cleanup step is ever required after an unclean shutdown.
|
|
- New direct dependency: `filelock`, added to `apps/device-host-agent/pyproject.toml` (already present transitively in `uv.lock` today; see design.md for why a maintained cross-platform library was chosen over hand-rolled `fcntl`/`msvcrt` branching for this specific mechanism).
|
|
|
|
## Capabilities
|
|
|
|
### New Capabilities
|
|
- `host-agent-single-instance-lock`: a local, per-installation exclusive-execution guarantee that prevents two Host Agent processes sharing the same `identity_path` state directory from running — and participating in the claim/heartbeat protocol — concurrently.
|
|
|
|
### Modified Capabilities
|
|
(none — `host-agent-protocol`'s outbound protocol behavior is unchanged; this change only gates whether a second local process is ever allowed to begin participating in that protocol)
|
|
|
|
## Impact
|
|
|
|
- Affected code: `apps/device-host-agent/host_agent/app.py` (`create_application()`/`HostAgentApplication` wiring), a new `apps/device-host-agent/host_agent/instance_lock.py` module, `apps/device-host-agent/pyproject.toml` (new `filelock` dependency).
|
|
- Not affected: `cloud.*`, `apps/cloud-api`, the outbound claim/heartbeat/lease protocol itself, `device/manager.py`, `driver/*`.
|
|
- Operational impact: an operator or supervisor that unintentionally starts a second instance now gets an immediate, clear local failure at startup instead of a delayed, confusing `DeviceNotFoundError` surfacing minutes later on an unrelated task dispatch. A legitimate restart (old process fully exited before the new one starts) is unaffected, since the lock releases on exit.
|