agentic-mobile-control/openspec/changes/task-cancellation/design.md

Context

Task cancellation has no implementation anywhere in the stack today:

• core/models.py's TaskStatus reserves "cancelled" but no code path assigns it. runtime/task.py::TaskRunner._interrupt_task() marks should_stop-triggered interruption as status="failed", failure_reason="execution interrupted" — this is the only existing consumer of should_stop, and it conflates "lease was lost" with "user asked to stop" (both produce failed).
• cloud/scheduler.py's ScheduledTaskStatus is queued | assigned | dispatched | done | failed — no cancelled, no cancel operation on TaskScheduler or CloudRepository.
• The internal Host↔Cloud protocol (cloud/internal_api/models.py) has exactly one channel from Cloud back to Host during execution: LeaseRenewalResponse, currently {status: Literal["renewed"], lease_expires_at: datetime}. There is no push channel; the Host Agent is purely outbound (heartbeat, claim long-poll, renew, report result).
• workflow/ already has the target shape: WorkflowRunStatus includes cancelled, WorkflowRunner._drive() checks should_stop() at each step boundary and calls self._checkpoint(run, ..., "cancelled"). This design generalizes that pattern to TaskRunner/ScheduledTask rather than inventing a new one.
• Three prior changes (cloud-console, cloud-control-plane-integration, cloud-console-governance) explicitly deferred cancellation, citing "no scheduler/repository operation for this exists." This change adds that operation.

Existing collaborative-stop infrastructure this design reuses instead of replacing: ActiveAssignmentRunner.run() (apps/device-host-agent/host_agent/lease.py) races execution against a renewal loop; the renewal loop is the only place the Host Agent talks to Cloud while a task is running. should_stop already flows ActiveAssignmentRunner → AssignmentExecutor.execute() → TaskRunner.run()/WorkflowRunner.run(), checked at step boundaries (never mid-step).

Goals / Non-Goals

Goals:

• Let an authorized caller cancel a task in queued, assigned, or dispatched state through the public SDK.
• For queued tasks (never dispatched to a Host), cancellation is synchronous and immediate — no Host round-trip needed.
• For assigned/dispatched tasks (a Host may be actively executing), cancellation is collaborative: the Host learns about it at its next lease renewal (≤ ~1/3 of lease_duration_seconds, matching the existing lease-loss detection latency) and stops at the next step boundary, exactly like a lost lease does today.
• Make cancelled a real, reachable terminal state end-to-end: core/models.py's TaskStatus, cloud/scheduler.py's ScheduledTaskStatus, the Cloud Console, and the Host Agent local console.
• Distinguish "cancelled by request" from "failed" in every layer that currently reports should_stop-triggered stops as generic failure, so operators can tell the two apart.
• Cancellation is idempotent: cancelling an already-cancelled or already-terminal task is a no-op with a clear response, not an error that implies something changed.

Non-Goals:

• No mid-step interruption. A single in-flight action (a tap, an LLM planning call) is never aborted mid-flight; the existing step-boundary granularity of should_stop is unchanged. A slow single step still finishes before a dispatched task's cancellation takes effect.
• No forced/instant kill of a Host Agent process or its OS-level subprocess tree. This is cooperative cancellation only, consistent with the project's existing lease-loss handling — not a new capability class.
• No per-task ownership/ACL model. The Cloud repository does not track which principal submitted a task; cancel authorization reuses the existing tasks:submit scope (whoever can submit a task can cancel any task). Adding submitter-scoped authorization is a distinct governance change, out of scope here (parallels cloud-console-governance's existing target-based, not ownership-based, model).
• No retry-from-UI or task editing. Only stopping a task early; resubmission remains a separate, already-existing "submit a new task" action.
• No cancellation of individual workflow steps independent of the whole run; workflow cancellation continues to mean "stop the run," which workflow/ already implements.
• No change to TaskDispatcher (cloud/dispatch.py) — confirmed dead/dev-only code per prior investigation; not worth extending for cancellation.

Decisions

D1: LeaseRenewalResponse.cancel_requested: bool is the only new wire signal

Rejected alternatives: (a) a new dedicated Host-polled "check cancellation" endpoint — adds a second polling loop and a second latency bound to reason about, when renewal already runs on a well-understood cadence; (b) a push/webhook mechanism — breaks the explicit "Host Agent operation requires no inbound cloud connection" requirement in host-agent-protocol's spec, which is a hard architectural constraint, not just current practice.

cancel_requested defaults to False. When the Cloud repository's renew_lease finds the active ScheduledTask has a pending cancellation record, it still renews the lease normally (a task must keep a live lease while the Host works through shutdown) but flags cancel_requested=True in the response. The Host Agent's _renew_while_running loop treats this the same way it treats StaleLeaseError: mark the LeaseGuard (extended with a distinct reason string, e.g. "cancellation requested by control plane") and let the existing should_stop composite (guard.is_lost() or self._stop_requested.is_set()) do the rest — no new boolean plumbed through ActiveAssignmentRunner/AssignmentExecutor signatures.

D2: Cancellation request is durable state, not a fire-and-forget signal

A POST .../cancel on an assigned/dispatched task writes a cancel_requested_at timestamp (see D5 schema) rather than only flipping an in-memory flag or directly setting status="cancelled". Rationale: the Host may not renew again for up to lease_duration_seconds / 3; if the Cloud process restarts in that window, an in-memory signal would be lost silently, defeating the cancellation with no user-visible error. A durable row survives restarts and is what renew_lease reads.

The scheduled task's status only transitions to cancelled once the Host has actually stopped and reported it (D3) — or, for the immediate queued case, synchronously in the same request. This means assigned/dispatched tasks pass through an intermediate, observable "cancellation pending" state (surfaced to callers as the existing status value plus a non-null cancel_requested_at, not a new status literal — see D6) rather than jumping straight to cancelled before the Host has confirmed.

D3: Host reports cancelled as a new terminal outcome via the existing result-report endpoint

TerminalResultRequest.status (internal_api/models.py) is Literal["done", "failed"]. This design adds "cancelled" to that literal rather than inventing a parallel cancellation-report endpoint, because report_result's idempotency/lease-validation logic (record_task_result in sql_repository.py) already handles exactly the concurrency shape needed (attempt/lease-id conflict checks, already_recorded replay safety) — duplicating it for a cancel-specific endpoint would be pure risk with no benefit. AssignmentExecutionResult.status (host_agent/assignment.py) similarly gains a "cancelled" value alongside "done"/"failed", set when TaskRunner.run() / WorkflowRunner.run() returns because of a cancellation-flavored stop rather than a lost lease or genuine step failure (see D4).

D4: should_stop becomes a richer signal than a bare boolean at the TaskRunner boundary

Today StopRequested = Callable[[], bool]. Distinguishing "cancelled" from "lease lost"/"shutdown requested" only matters for the reason recorded on the terminal status — the control-flow behavior (stop at the next step boundary) is identical. Rather than changing the should_stop callable's signature (which would ripple through workflow/runner.py, tests, and every caller), TaskRunner._interrupt_task() and WorkflowRunner._drive()'s stop branch gain an optional second callable, stop_reason: Callable[[], str] | None, defaulting to the existing generic "execution interrupted" string when absent. ActiveAssignmentRunner supplies a stop_reason that reads LeaseGuard.reason (already a string field) so "cancellation requested by control plane" vs "lease rejected by control plane" flows through unchanged plumbing. TaskRunner._interrupt_task() sets status="cancelled" when the reason string indicates cancellation, else keeps status="failed" (lease loss remains a failure, not a cancellation, matching current behavior for that case). This is the narrowest change that gets a real cancelled status without touching every should_stop-typed parameter across the codebase.

Alternative considered and rejected: add a third CancelRequested callable parallel to should_stop. Rejected because it doubles the number of callables threaded through ActiveAssignmentRunner → AssignmentExecutor → TaskRunner/WorkflowRunner for information that's only needed at the one moment execution actually stops.

D5: Schema — one new nullable column pair on scheduled_tasks, no new table

packages/cloud-platform/cloud/migrations/versions/0012_task_cancellation.py adds to ScheduledTaskRow/scheduled_tasks:

• cancel_requested_at: str | None (ISO datetime, nullable) — set when a cancel request is recorded against an assigned/dispatched task; cleared (NULL) when the task reaches any terminal status, so a subsequent, different attempt of the same task id (if retry-on-lease-expiry logic in reap_expired_leases requeues it) doesn't inherit a stale cancellation.

No new table: the volume and access pattern (one pending value per task, read on every renewal, written once per cancel call) don't justify the join/joinless-read trade-off a separate task_cancellation_requests table would add, and there is exactly one prior migration precedent for this shape — 0008_task_progress_columns added nullable scalar columns directly to scheduled_tasks for the same reason (frequently-read, single-value-per-task state). TaskAttemptRow/task_attempts needs no schema change: its status column already accepts free-form strings and gains "cancelled" as a value alongside "assigned"/"dispatched"/"expired"/"done"/"failed".

CloudRepository Protocol gains:

• request_task_cancellation(task_id, *, requested_at) -> CancellationRequestStatus where CancellationRequestStatus = Literal["requested", "already_terminal", "already_requested", "not_found"] — synchronously transitions a queued task straight to cancelled (there's no Host attempt in flight to notify) and otherwise sets cancel_requested_at on an assigned/dispatched task.
• renew_lease gains a cancel_requested boolean in its return path (or the caller re-reads the row — implementation detail left to tasks.md) so internal_api/api.py's renew_assignment handler can populate LeaseRenewalResponse.cancel_requested.
• record_task_result accepts status: Literal["done", "failed", "cancelled"] (widened from today's TerminalTaskStatus = Literal["done", "failed"]) and clears cancel_requested_at on write.

D6: No new ScheduledTaskStatus value for "cancellation pending"

Considered adding "cancelling" as a distinct status between assigned/dispatched and cancelled. Rejected: it would ripple into every place that already pattern-matches the existing five-value ScheduledTaskStatus (scheduler assignment logic, device reservation/list_reserved_device_ids, Cloud Console status filter, SDK response Literal), each needing to decide whether "cancelling" behaves like "assigned" (device still reserved, task still excluded from re-assignment) — which it always would, making the new status a strict synonym with an extra bit of information. Instead, "cancellation pending" is expressed as status="assigned" (or "dispatched") plus non-null cancel_requested_at — every existing status-based code path (assignment matching, device reservation, retry-on-expiry) keeps working unmodified, and callers that want to show "cancelling…" in a UI check the extra field.

D7: Public cancel endpoint shape

POST /v1/tasks/{task_id}/cancel, 202 Accepted for the pending (assigned/dispatched) case and 200 OK for the immediate (queued) case, both returning a small TaskCancellationResponse {task_id, status} reflecting the resulting ScheduledTask status. Repeated calls against an already-cancelled or already-cancel_requested_at-set task return the same response idempotently (HTTP 200, not a conflict) — matching the project's established idempotency style for report_result/renew_assignment, which return already_recorded/success on replay rather than erroring. Calling cancel on a task already done/failed returns 409 Conflict with a clear "task is already terminal" detail, mirroring _stale_lease_conflict's existing shape.

Risks / Trade-offs

• [Operators expect cancellation to be instant] → It isn't, by design (D1's latency bound). The Cloud Console surfaces the cancel_requested_at pending state distinctly (D6) so operators see "cancellation requested" rather than a UI that looks stuck; docs (docs/CLOUD_DEPLOYMENT.md) get an explicit latency note per the proposal's Impact section.
• [A Host Agent that never renews again (already offline, or wedged past its lease) never learns about the cancellation] → Existing reap_expired_leases already requeues or fails such tasks once the lease expires; this change doesn't need new machinery for that case — an offline Host's task eventually reaches a terminal state via the existing reaper, at which point cancel_requested_at being non-null is irrelevant (task is already terminal). Worth noting in tasks.md that the reaper's requeue path should NOT requeue (re-assign) a task with cancel_requested_at set — it should mark it cancelled instead of queued, or the cancellation would be silently dropped on retry.
• [tasks:submit-scoped cancel with no per-task ownership means any authorized submitter can cancel any other submitter's task] → Accepted for this change (Non-Goals); this matches the existing coarse-grained scope model everywhere else in the SDK (task reading is likewise not ownership-scoped) and narrowing it is cloud-console-governance territory, not this change's.
• [Widening TaskStatus/ScheduledTaskStatus/TaskAttemptRow.status literals is a backward-compatible additive change in Python, but any exhaustive switch/discriminated union in the TypeScript Console (types.ts's TaskStatus) will fail to compile until updated] → Caught at Console build time (Vite/vue-tsc), not runtime; tasks.md includes updating cloud-console/src/types.ts and any exhaustiveness-checked switch in the same commit as the backend change.
• [Race: cancel request arrives between the Host's claim_assignment (queued→dispatched transition happens inside claim_assignment, not assign_task) and its first renewal] → Already handled by D2's durable-row design: the cancel call checks current status regardless of exactly when the Host claims, and renew_lease always re-reads the current row, so there's no window where the signal is lost — only a window (bounded by D1's latency) where it hasn't been observed yet.

Migration Plan

1. Ship the Alembic migration (0012) — additive nullable column, no backfill needed, safe to apply with the application running (existing tasks get cancel_requested_at = NULL, behaviorally identical to today).
2. Deploy Cloud API with the new repository methods, internal renew_assignment response field (cancel_requested, defaults False — old Host Agents ignore unknown fields), and the new public cancel endpoint. This step alone is a no-op for existing Hosts: nothing calls the new endpoint yet, and LeaseRenewalResponse gaining an optional field is backward-compatible with any Host Agent version already deployed (Pydantic response models are additive-safe for JSON-decoding clients that only read known fields).
3. Deploy Host Agents with the updated lease.py/assignment.py/client.py that read and act on cancel_requested. Hosts not yet updated simply never observe cancellation requests (task stays "pending cancellation" until its lease naturally expires and the reaper marks it cancelled per the Risks section) — a soft-fail, not a hard error.
4. Ship the Cloud Console and Host Agent local console UI changes last, once the backend contract is stable.
5. Rollback: the migration is purely additive and safe to leave in place even if the feature is disabled; no rollback migration is required beyond the standard Alembic downgrade (drop the column) if a full revert is ever needed.

Open Questions

• Whether the public cancel endpoint should also accept an optional caller-supplied reason string (for audit/UI display) — left to tasks.md to decide during implementation; not a design-blocking question since it's purely additive to TaskCancellationRequest if added later.
• Whether reap_expired_leases' "mark cancelled instead of requeue when cancel_requested_at is set" behavior (noted in Risks) needs its own explicit requirement in the task-scheduler delta spec or can be covered as an implementation detail of the existing restart-recovery requirement — resolve when writing specs.md.