Proposal, design, spec, and tasks for the per-installation exclusive instance lock. 15/16 tasks complete; only manual real-environment verification (5.4) remains, with semantics covered by unit tests in test_app.py and test_instance_lock.py. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
4.0 KiB
Why
A production task dispatch failed with DeviceNotFoundError: unknown device device-8967d09f0d5f4cf49f7361a9f0dcb0ce (host host-311f68..., 2026-07-14) even though the Host Agent's own console showed that same device as connected. Root-cause tracing confirmed: DeviceManager (device/manager.py) is in-process, in-memory, and per-process, with no persistence and no reactive removal. TaskScheduler.assign() (cloud/scheduler.py) dispatches to a host_id based on a periodically-synced DevicePool snapshot, with no concept of which OS process is currently answering that host's long-poll. If two Host Agent processes ever run concurrently under the same host identity — e.g. a supervisor starting a replacement before a hung previous instance has fully exited, or an operator accidentally leaving a second instance running in another terminal/tmux pane — both heartbeat and both claim under the same host_id, and a claim can land on whichever instance answers next, including one whose own in-memory DeviceManager never registered the device. Nothing in the codebase today prevents this: a repo-wide search confirms no pidfile, lockfile, or flock-style guard exists anywhere in apps/device-host-agent, and both identity_path and the device-config store default to process-working-directory-relative paths, which makes an accidental duplicate launch easy to trigger unnoticed.
What Changes
create_application()(apps/device-host-agent/host_agent/app.py:145) acquires a local, exclusive, non-blocking instance lock as its first action, beforeresolve_host_identity()or any device-enrollment network call.- The lock file is colocated with the existing per-installation state directory (
identity_path.parent— the same directoryapp.pyalready uses forhost_console_history.sqlite3andhost_governance_policy.json), so this requires no new configuration surface. - If the lock is already held by a live process, the Host Agent logs a clear, actionable "duplicate instance" error (distinct from a generic startup failure) and exits non-zero without contacting the control plane, enrolling devices, or starting the console/heartbeat/claim loops.
- The lock releases automatically on process exit for any reason — clean shutdown, unhandled exception, or a forced kill — via OS-level advisory file locking rather than a pidfile/sentinel-existence check, so no stale-lock cleanup step is ever required after an unclean shutdown.
- New direct dependency:
filelock, added toapps/device-host-agent/pyproject.toml(already present transitively inuv.locktoday; see design.md for why a maintained cross-platform library was chosen over hand-rolledfcntl/msvcrtbranching for this specific mechanism).
Capabilities
New Capabilities
host-agent-single-instance-lock: a local, per-installation exclusive-execution guarantee that prevents two Host Agent processes sharing the sameidentity_pathstate directory from running — and participating in the claim/heartbeat protocol — concurrently.
Modified Capabilities
(none — host-agent-protocol's outbound protocol behavior is unchanged; this change only gates whether a second local process is ever allowed to begin participating in that protocol)
Impact
- Affected code:
apps/device-host-agent/host_agent/app.py(create_application()/HostAgentApplicationwiring), a newapps/device-host-agent/host_agent/instance_lock.pymodule,apps/device-host-agent/pyproject.toml(newfilelockdependency). - Not affected:
cloud.*,apps/cloud-api, the outbound claim/heartbeat/lease protocol itself,device/manager.py,driver/*. - Operational impact: an operator or supervisor that unintentionally starts a second instance now gets an immediate, clear local failure at startup instead of a delayed, confusing
DeviceNotFoundErrorsurfacing minutes later on an unrelated task dispatch. A legitimate restart (old process fully exited before the new one starts) is unaffected, since the lock releases on exit.