agentic-mobile-control/openspec/changes/host-agent-single-instance-lock/proposal.md at master

Files

T

q792602257andClaude Sonnet 5 bb6f24bbcb docs(openspec): correct host-agent-single-instance-lock's Why section

This change was originally proposed as the root-cause fix for the
2026-07-14 DeviceNotFoundError incident. That diagnosis was wrong: it was
subsequently confirmed only one Host Agent process was running at the time,
ruling out the duplicate-process precondition this change addresses. The
actual root cause was execution.py's create_task_runner() omitting manager=
when wiring TaskRunner (see 08cef7c). Reframe the Why section: this change
stands on its own as independent duplicate-process hardening, not as a fix
for an incident it turned out not to have caused.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

2026-07-14 16:36:04 +08:00

4.8 KiB

Raw Permalink Blame History

Why

Duplicate Host Agent processes sharing the same host identity are a real, unguarded risk: a repo-wide search confirms no pidfile, lockfile, or flock-style guard exists anywhere in apps/device-host-agent, and both identity_path and the device-config store default to process-working-directory-relative paths, which makes an accidental duplicate launch easy to trigger unnoticed (e.g. a supervisor starting a replacement before a hung previous instance has fully exited, or an operator accidentally leaving a second instance running in another terminal/tmux pane). If it ever happens, the failure mode is severe and confusing: DeviceManager (device/manager.py) is in-process, in-memory, and per-process, with no persistence and no reactive removal, while TaskScheduler.assign() (cloud/scheduler.py) dispatches to a host_id based on a periodically-synced DevicePool snapshot with no concept of which OS process is currently answering that host's long-poll — both instances would heartbeat and claim under the same host_id, and a claim could land on whichever instance answers next, including one whose own in-memory DeviceManager never registered the device.

This change was originally proposed as the root-cause fix for a production incident (DeviceNotFoundError: unknown device device-8967d09f0d5f4cf49f7361a9f0dcb0ce, host host-311f68..., 2026-07-14). That diagnosis was wrong: it was subsequently confirmed that only one Host Agent process was running at the time, which rules out the duplicate-process precondition this change addresses. The actual root cause was create_task_runner() (apps/device-host-agent/host_agent/execution.py) omitting manager= when wiring TaskRunner's observer/screenshot_provider, so both silently fell back to the process-global DEFAULT_MANAGER singleton instead of the Host Agent's real, device-populated DeviceManager — fixed directly in code, without an OpenSpec proposal, since it was a small, fully-diagnosed, single-file wiring bug. This proposal is kept anyway, on its own merits: duplicate-process protection is a legitimate, independent hardening measure against a real gap the investigation surfaced along the way, not a fix for an incident it turned out not to have caused.

What Changes

create_application() (apps/device-host-agent/host_agent/app.py:145) acquires a local, exclusive, non-blocking instance lock as its first action, before resolve_host_identity() or any device-enrollment network call.
The lock file is colocated with the existing per-installation state directory (identity_path.parent — the same directory app.py already uses for host_console_history.sqlite3 and host_governance_policy.json), so this requires no new configuration surface.
If the lock is already held by a live process, the Host Agent logs a clear, actionable "duplicate instance" error (distinct from a generic startup failure) and exits non-zero without contacting the control plane, enrolling devices, or starting the console/heartbeat/claim loops.
The lock releases automatically on process exit for any reason — clean shutdown, unhandled exception, or a forced kill — via OS-level advisory file locking rather than a pidfile/sentinel-existence check, so no stale-lock cleanup step is ever required after an unclean shutdown.
New direct dependency: filelock, added to apps/device-host-agent/pyproject.toml (already present transitively in uv.lock today; see design.md for why a maintained cross-platform library was chosen over hand-rolled fcntl/msvcrt branching for this specific mechanism).

Capabilities

New Capabilities

host-agent-single-instance-lock: a local, per-installation exclusive-execution guarantee that prevents two Host Agent processes sharing the same identity_path state directory from running — and participating in the claim/heartbeat protocol — concurrently.

Modified Capabilities

(none — host-agent-protocol's outbound protocol behavior is unchanged; this change only gates whether a second local process is ever allowed to begin participating in that protocol)

Impact

Affected code: apps/device-host-agent/host_agent/app.py (create_application()/HostAgentApplication wiring), a new apps/device-host-agent/host_agent/instance_lock.py module, apps/device-host-agent/pyproject.toml (new filelock dependency).
Not affected: cloud.*, apps/cloud-api, the outbound claim/heartbeat/lease protocol itself, device/manager.py, driver/*.
Operational impact: an operator or supervisor that unintentionally starts a second instance now gets an immediate, clear local failure at startup instead of a delayed, confusing DeviceNotFoundError surfacing minutes later on an unrelated task dispatch. A legitimate restart (old process fully exited before the new one starts) is unaffected, since the lock releases on exit.

4.8 KiB Raw Permalink Blame History

Why