## Context Task cancellation has no implementation anywhere in the stack today: - `core/models.py`'s `TaskStatus` reserves `"cancelled"` but no code path assigns it. `runtime/task.py::TaskRunner._interrupt_task()` marks `should_stop`-triggered interruption as `status="failed", failure_reason="execution interrupted"` — this is the only existing consumer of `should_stop`, and it conflates "lease was lost" with "user asked to stop" (both produce `failed`). - `cloud/scheduler.py`'s `ScheduledTaskStatus` is `queued | assigned | dispatched | done | failed` — no `cancelled`, no cancel operation on `TaskScheduler` or `CloudRepository`. - The internal Host↔Cloud protocol (`cloud/internal_api/models.py`) has exactly one channel from Cloud back to Host during execution: `LeaseRenewalResponse`, currently `{status: Literal["renewed"], lease_expires_at: datetime}`. There is no push channel; the Host Agent is purely outbound (heartbeat, claim long-poll, renew, report result). - `workflow/` already has the target shape: `WorkflowRunStatus` includes `cancelled`, `WorkflowRunner._drive()` checks `should_stop()` at each step boundary and calls `self._checkpoint(run, ..., "cancelled")`. This design generalizes that pattern to `TaskRunner`/`ScheduledTask` rather than inventing a new one. - Three prior changes (`cloud-console`, `cloud-control-plane-integration`, `cloud-console-governance`) explicitly deferred cancellation, citing "no scheduler/repository operation for this exists." This change adds that operation. Existing collaborative-stop infrastructure this design reuses instead of replacing: `ActiveAssignmentRunner.run()` (`apps/device-host-agent/host_agent/lease.py`) races execution against a renewal loop; the renewal loop is the only place the Host Agent talks to Cloud while a task is running. `should_stop` already flows `ActiveAssignmentRunner` → `AssignmentExecutor.execute()` → `TaskRunner.run()`/`WorkflowRunner.run()`, checked at step boundaries (never mid-step). ## Goals / Non-Goals **Goals:** - Let an authorized caller cancel a task in `queued`, `assigned`, or `dispatched` state through the public SDK. - For `queued` tasks (never dispatched to a Host), cancellation is synchronous and immediate — no Host round-trip needed. - For `assigned`/`dispatched` tasks (a Host may be actively executing), cancellation is collaborative: the Host learns about it at its next lease renewal (≤ ~1/3 of `lease_duration_seconds`, matching the existing lease-loss detection latency) and stops at the next step boundary, exactly like a lost lease does today. - Make `cancelled` a real, reachable terminal state end-to-end: `core/models.py`'s `TaskStatus`, `cloud/scheduler.py`'s `ScheduledTaskStatus`, the Cloud Console, and the Host Agent local console. - Distinguish "cancelled by request" from "failed" in every layer that currently reports `should_stop`-triggered stops as generic failure, so operators can tell the two apart. - Cancellation is idempotent: cancelling an already-cancelled or already-terminal task is a no-op with a clear response, not an error that implies something changed. **Non-Goals:** - No mid-step interruption. A single in-flight action (a tap, an LLM planning call) is never aborted mid-flight; the existing step-boundary granularity of `should_stop` is unchanged. A slow single step still finishes before a `dispatched` task's cancellation takes effect. - No forced/instant kill of a Host Agent process or its OS-level subprocess tree. This is cooperative cancellation only, consistent with the project's existing lease-loss handling — not a new capability class. - No per-task ownership/ACL model. The Cloud repository does not track which principal submitted a task; cancel authorization reuses the existing `tasks:submit` scope (whoever can submit a task can cancel any task). Adding submitter-scoped authorization is a distinct governance change, out of scope here (parallels `cloud-console-governance`'s existing target-based, not ownership-based, model). - No retry-from-UI or task editing. Only stopping a task early; resubmission remains a separate, already-existing "submit a new task" action. - No cancellation of individual workflow steps independent of the whole run; workflow cancellation continues to mean "stop the run," which `workflow/` already implements. - No change to `TaskDispatcher` (`cloud/dispatch.py`) — confirmed dead/dev-only code per prior investigation; not worth extending for cancellation. ## Decisions ### D1: `LeaseRenewalResponse.cancel_requested: bool` is the only new wire signal Rejected alternatives: (a) a new dedicated Host-polled "check cancellation" endpoint — adds a second polling loop and a second latency bound to reason about, when renewal already runs on a well-understood cadence; (b) a push/webhook mechanism — breaks the explicit "Host Agent operation requires no inbound cloud connection" requirement in `host-agent-protocol`'s spec, which is a hard architectural constraint, not just current practice. `cancel_requested` defaults to `False`. When the Cloud repository's `renew_lease` finds the active `ScheduledTask` has a pending cancellation record, it still renews the lease normally (a task must keep a live lease while the Host works through shutdown) but flags `cancel_requested=True` in the response. The Host Agent's `_renew_while_running` loop treats this the same way it treats `StaleLeaseError`: mark the `LeaseGuard` (extended with a distinct reason string, e.g. `"cancellation requested by control plane"`) and let the existing `should_stop` composite (`guard.is_lost() or self._stop_requested.is_set()`) do the rest — no new boolean plumbed through `ActiveAssignmentRunner`/`AssignmentExecutor` signatures. ### D2: Cancellation request is durable state, not a fire-and-forget signal A `POST .../cancel` on an `assigned`/`dispatched` task writes a `cancel_requested_at` timestamp (see D5 schema) rather than only flipping an in-memory flag or directly setting `status="cancelled"`. Rationale: the Host may not renew again for up to `lease_duration_seconds / 3`; if the Cloud process restarts in that window, an in-memory signal would be lost silently, defeating the cancellation with no user-visible error. A durable row survives restarts and is what `renew_lease` reads. The scheduled task's `status` only transitions to `cancelled` once the Host has actually stopped and reported it (D3) — or, for the immediate `queued` case, synchronously in the same request. This means `assigned`/`dispatched` tasks pass through an intermediate, observable "cancellation pending" state (surfaced to callers as the existing `status` value plus a non-null `cancel_requested_at`, not a new status literal — see D6) rather than jumping straight to `cancelled` before the Host has confirmed. ### D3: Host reports `cancelled` as a new terminal outcome via the existing result-report endpoint `TerminalResultRequest.status` (`internal_api/models.py`) is `Literal["done", "failed"]`. This design adds `"cancelled"` to that literal rather than inventing a parallel cancellation-report endpoint, because `report_result`'s idempotency/lease-validation logic (`record_task_result` in `sql_repository.py`) already handles exactly the concurrency shape needed (attempt/lease-id conflict checks, `already_recorded` replay safety) — duplicating it for a cancel-specific endpoint would be pure risk with no benefit. `AssignmentExecutionResult.status` (`host_agent/assignment.py`) similarly gains a `"cancelled"` value alongside `"done"`/`"failed"`, set when `TaskRunner.run()` / `WorkflowRunner.run()` returns because of a cancellation-flavored stop rather than a lost lease or genuine step failure (see D4). ### D4: `should_stop` becomes a richer signal than a bare boolean at the `TaskRunner` boundary Today `StopRequested = Callable[[], bool]`. Distinguishing "cancelled" from "lease lost"/"shutdown requested" only matters for the *reason recorded on the terminal status* — the control-flow behavior (stop at the next step boundary) is identical. Rather than changing the `should_stop` callable's signature (which would ripple through `workflow/runner.py`, tests, and every caller), `TaskRunner._interrupt_task()` and `WorkflowRunner._drive()`'s stop branch gain an optional second callable, `stop_reason: Callable[[], str] | None`, defaulting to the existing generic `"execution interrupted"` string when absent. `ActiveAssignmentRunner` supplies a `stop_reason` that reads `LeaseGuard.reason` (already a string field) so `"cancellation requested by control plane"` vs `"lease rejected by control plane"` flows through unchanged plumbing. `TaskRunner._interrupt_task()` sets `status="cancelled"` when the reason string indicates cancellation, else keeps `status="failed"` (lease loss remains a failure, not a cancellation, matching current behavior for that case). This is the narrowest change that gets a real `cancelled` status without touching every `should_stop`-typed parameter across the codebase. Alternative considered and rejected: add a third `CancelRequested` callable parallel to `should_stop`. Rejected because it doubles the number of callables threaded through `ActiveAssignmentRunner → AssignmentExecutor → TaskRunner/WorkflowRunner` for information that's only needed at the one moment execution actually stops. ### D5: Schema — one new nullable column pair on `scheduled_tasks`, no new table `packages/cloud-platform/cloud/migrations/versions/0012_task_cancellation.py` adds to `ScheduledTaskRow`/`scheduled_tasks`: - `cancel_requested_at: str | None` (ISO datetime, nullable) — set when a cancel request is recorded against an `assigned`/`dispatched` task; cleared (`NULL`) when the task reaches any terminal status, so a subsequent, different attempt of the same task id (if retry-on-lease-expiry logic in `reap_expired_leases` requeues it) doesn't inherit a stale cancellation. No new table: the volume and access pattern (one pending value per task, read on every renewal, written once per cancel call) don't justify the join/joinless-read trade-off a separate `task_cancellation_requests` table would add, and there is exactly one prior migration precedent for this shape — `0008_task_progress_columns` added nullable scalar columns directly to `scheduled_tasks` for the same reason (frequently-read, single-value-per-task state). `TaskAttemptRow`/`task_attempts` needs no schema change: its `status` column already accepts free-form strings and gains `"cancelled"` as a value alongside `"assigned"/"dispatched"/"expired"/"done"/"failed"`. `CloudRepository` Protocol gains: - `request_task_cancellation(task_id, *, requested_at) -> CancellationRequestStatus` where `CancellationRequestStatus = Literal["requested", "already_terminal", "already_requested", "not_found"]` — synchronously transitions a `queued` task straight to `cancelled` (there's no Host attempt in flight to notify) and otherwise sets `cancel_requested_at` on an `assigned`/`dispatched` task. - `renew_lease` gains a `cancel_requested` boolean in its return path (or the caller re-reads the row — implementation detail left to tasks.md) so `internal_api/api.py`'s `renew_assignment` handler can populate `LeaseRenewalResponse.cancel_requested`. - `record_task_result` accepts `status: Literal["done", "failed", "cancelled"]` (widened from today's `TerminalTaskStatus = Literal["done", "failed"]`) and clears `cancel_requested_at` on write. ### D6: No new `ScheduledTaskStatus` value for "cancellation pending" Considered adding `"cancelling"` as a distinct status between `assigned`/`dispatched` and `cancelled`. Rejected: it would ripple into every place that already pattern-matches the existing five-value `ScheduledTaskStatus` (scheduler assignment logic, device reservation/`list_reserved_device_ids`, Cloud Console status filter, SDK response `Literal`), each needing to decide whether "cancelling" behaves like "assigned" (device still reserved, task still excluded from re-assignment) — which it always would, making the new status a strict synonym with an extra bit of information. Instead, "cancellation pending" is expressed as `status="assigned"` (or `"dispatched"`) plus non-null `cancel_requested_at` — every existing status-based code path (assignment matching, device reservation, retry-on-expiry) keeps working unmodified, and callers that want to show "cancelling…" in a UI check the extra field. ### D7: Public cancel endpoint shape `POST /v1/tasks/{task_id}/cancel`, `202 Accepted` for the pending (assigned/dispatched) case and `200 OK` for the immediate (queued) case, both returning a small `TaskCancellationResponse {task_id, status}` reflecting the resulting `ScheduledTask` status. Repeated calls against an already-cancelled or already-`cancel_requested_at`-set task return the same response idempotently (HTTP 200, not a conflict) — matching the project's established idempotency style for `report_result`/`renew_assignment`, which return `already_recorded`/success on replay rather than erroring. Calling cancel on a task already `done`/`failed` returns `409 Conflict` with a clear "task is already terminal" detail, mirroring `_stale_lease_conflict`'s existing shape. ## Risks / Trade-offs - [Operators expect cancellation to be instant] → It isn't, by design (D1's latency bound). The Cloud Console surfaces the `cancel_requested_at` pending state distinctly (D6) so operators see "cancellation requested" rather than a UI that looks stuck; docs (`docs/CLOUD_DEPLOYMENT.md`) get an explicit latency note per the proposal's Impact section. - [A Host Agent that never renews again (already offline, or wedged past its lease) never learns about the cancellation] → Existing `reap_expired_leases` already requeues or fails such tasks once the lease expires; this change doesn't need new machinery for that case — an offline Host's task eventually reaches a terminal state via the existing reaper, at which point `cancel_requested_at` being non-null is irrelevant (task is already terminal). Worth noting in tasks.md that the reaper's requeue path should NOT requeue (re-assign) a task with `cancel_requested_at` set — it should mark it `cancelled` instead of `queued`, or the cancellation would be silently dropped on retry. - [`tasks:submit`-scoped cancel with no per-task ownership means any authorized submitter can cancel any other submitter's task] → Accepted for this change (Non-Goals); this matches the existing coarse-grained scope model everywhere else in the SDK (task reading is likewise not ownership-scoped) and narrowing it is `cloud-console-governance` territory, not this change's. - [Widening `TaskStatus`/`ScheduledTaskStatus`/`TaskAttemptRow.status` literals is a backward-compatible additive change in Python, but any exhaustive `switch`/discriminated union in the TypeScript Console (`types.ts`'s `TaskStatus`) will fail to compile until updated] → Caught at Console build time (Vite/`vue-tsc`), not runtime; tasks.md includes updating `cloud-console/src/types.ts` and any exhaustiveness-checked switch in the same commit as the backend change. - [Race: cancel request arrives between the Host's `claim_assignment` (queued→dispatched transition happens inside `claim_assignment`, not `assign_task`) and its first renewal] → Already handled by D2's durable-row design: the cancel call checks current `status` regardless of exactly when the Host claims, and `renew_lease` always re-reads the current row, so there's no window where the signal is lost — only a window (bounded by D1's latency) where it hasn't been observed yet. ## Migration Plan 1. Ship the Alembic migration (0012) — additive nullable column, no backfill needed, safe to apply with the application running (existing tasks get `cancel_requested_at = NULL`, behaviorally identical to today). 2. Deploy Cloud API with the new repository methods, internal `renew_assignment` response field (`cancel_requested`, defaults `False` — old Host Agents ignore unknown fields), and the new public cancel endpoint. This step alone is a no-op for existing Hosts: nothing calls the new endpoint yet, and `LeaseRenewalResponse` gaining an optional field is backward-compatible with any Host Agent version already deployed (Pydantic response models are additive-safe for JSON-decoding clients that only read known fields). 3. Deploy Host Agents with the updated `lease.py`/`assignment.py`/`client.py` that read and act on `cancel_requested`. Hosts not yet updated simply never observe cancellation requests (task stays "pending cancellation" until its lease naturally expires and the reaper marks it `cancelled` per the Risks section) — a soft-fail, not a hard error. 4. Ship the Cloud Console and Host Agent local console UI changes last, once the backend contract is stable. 5. Rollback: the migration is purely additive and safe to leave in place even if the feature is disabled; no rollback migration is required beyond the standard Alembic downgrade (drop the column) if a full revert is ever needed. ## Open Questions - Whether the public cancel endpoint should also accept an optional caller-supplied reason string (for audit/UI display) — left to tasks.md to decide during implementation; not a design-blocking question since it's purely additive to `TaskCancellationRequest` if added later. - Whether `reap_expired_leases`' "mark cancelled instead of requeue when `cancel_requested_at` is set" behavior (noted in Risks) needs its own explicit requirement in the `task-scheduler` delta spec or can be covered as an implementation detail of the existing restart-recovery requirement — resolve when writing specs.md.