q792602257andClaude Opus 4.6
82567fd248
Tests / Test passed: 759
Proposal, design, spec, and tasks for the per-installation exclusive instance lock. 15/16 tasks complete; only manual real-environment verification (5.4) remains, with semantics covered by unit tests in test_app.py and test_instance_lock.py. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
26 lines
4.0 KiB
Markdown
26 lines
4.0 KiB
Markdown
## Why
|
|
|
|
A production task dispatch failed with `DeviceNotFoundError: unknown device device-8967d09f0d5f4cf49f7361a9f0dcb0ce` (host `host-311f68...`, 2026-07-14) even though the Host Agent's own console showed that same device as `connected`. Root-cause tracing confirmed: `DeviceManager` (`device/manager.py`) is in-process, in-memory, and per-process, with no persistence and no reactive removal. `TaskScheduler.assign()` (`cloud/scheduler.py`) dispatches to a `host_id` based on a periodically-synced `DevicePool` snapshot, with no concept of *which OS process* is currently answering that host's long-poll. If two Host Agent processes ever run concurrently under the same host identity — e.g. a supervisor starting a replacement before a hung previous instance has fully exited, or an operator accidentally leaving a second instance running in another terminal/tmux pane — both heartbeat and both claim under the same `host_id`, and a claim can land on whichever instance answers next, including one whose own in-memory `DeviceManager` never registered the device. Nothing in the codebase today prevents this: a repo-wide search confirms no pidfile, lockfile, or `flock`-style guard exists anywhere in `apps/device-host-agent`, and both `identity_path` and the device-config store default to process-working-directory-relative paths, which makes an accidental duplicate launch easy to trigger unnoticed.
|
|
|
|
## What Changes
|
|
|
|
- `create_application()` (`apps/device-host-agent/host_agent/app.py:145`) acquires a local, exclusive, non-blocking instance lock as its first action, before `resolve_host_identity()` or any device-enrollment network call.
|
|
- The lock file is colocated with the existing per-installation state directory (`identity_path.parent` — the same directory `app.py` already uses for `host_console_history.sqlite3` and `host_governance_policy.json`), so this requires no new configuration surface.
|
|
- If the lock is already held by a live process, the Host Agent logs a clear, actionable "duplicate instance" error (distinct from a generic startup failure) and exits non-zero without contacting the control plane, enrolling devices, or starting the console/heartbeat/claim loops.
|
|
- The lock releases automatically on process exit for any reason — clean shutdown, unhandled exception, or a forced kill — via OS-level advisory file locking rather than a pidfile/sentinel-existence check, so no stale-lock cleanup step is ever required after an unclean shutdown.
|
|
- New direct dependency: `filelock`, added to `apps/device-host-agent/pyproject.toml` (already present transitively in `uv.lock` today; see design.md for why a maintained cross-platform library was chosen over hand-rolled `fcntl`/`msvcrt` branching for this specific mechanism).
|
|
|
|
## Capabilities
|
|
|
|
### New Capabilities
|
|
- `host-agent-single-instance-lock`: a local, per-installation exclusive-execution guarantee that prevents two Host Agent processes sharing the same `identity_path` state directory from running — and participating in the claim/heartbeat protocol — concurrently.
|
|
|
|
### Modified Capabilities
|
|
(none — `host-agent-protocol`'s outbound protocol behavior is unchanged; this change only gates whether a second local process is ever allowed to begin participating in that protocol)
|
|
|
|
## Impact
|
|
|
|
- Affected code: `apps/device-host-agent/host_agent/app.py` (`create_application()`/`HostAgentApplication` wiring), a new `apps/device-host-agent/host_agent/instance_lock.py` module, `apps/device-host-agent/pyproject.toml` (new `filelock` dependency).
|
|
- Not affected: `cloud.*`, `apps/cloud-api`, the outbound claim/heartbeat/lease protocol itself, `device/manager.py`, `driver/*`.
|
|
- Operational impact: an operator or supervisor that unintentionally starts a second instance now gets an immediate, clear local failure at startup instead of a delayed, confusing `DeviceNotFoundError` surfacing minutes later on an unrelated task dispatch. A legitimate restart (old process fully exited before the new one starts) is unaffected, since the lock releases on exit.
|